Oct 30 18:43:26.768452 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 18:43:26.768466 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 18:43:26.768476 ip-10-0-136-46 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 18:43:26.768839 ip-10-0-136-46 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 18:43:36.866519 ip-10-0-136-46 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 18:43:36.866534 ip-10-0-136-46 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 170d6af1d49847f4a7057768f04adc63 -- Oct 30 18:45:17.979730 ip-10-0-136-46 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:45:18.459408 ip-10-0-136-46 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:45:18.459408 ip-10-0-136-46 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:45:18.459408 ip-10-0-136-46 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:45:18.459408 ip-10-0-136-46 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:45:18.465053 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.464966 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.465053 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465052 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.465053 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465057 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465062 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465065 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465068 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465071 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465074 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465076 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465079 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465082 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465084 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465087 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465090 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465092 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465096 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465099 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465101 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465104 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465107 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465109 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.465148 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465112 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465114 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465118 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465120 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465123 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465125 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465128 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465131 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465133 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465136 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465138 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465141 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465143 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465149 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465152 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465155 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465158 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465161 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465163 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.465635 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465166 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465169 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465171 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465174 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465176 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465179 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465181 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465184 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465187 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465190 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465193 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465195 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465198 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465200 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465203 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465206 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465209 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465211 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465214 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465217 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.466105 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465219 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465222 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465224 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465226 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465229 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465232 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465235 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465237 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465240 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465244 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465246 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465249 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465252 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465255 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465257 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465260 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465262 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465265 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465269 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.466630 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465274 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465278 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465281 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465284 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465286 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465289 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465295 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465297 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465300 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465302 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465305 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465308 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465311 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465314 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465316 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465319 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465322 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465324 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.467082 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465328 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465331 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465333 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465336 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465338 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465341 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.465343 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466134 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466141 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466144 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466147 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466153 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466155 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466158 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466168 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466171 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466174 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466177 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466179 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.467545 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466182 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466185 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466188 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466191 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466194 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466196 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466199 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466201 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466204 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466206 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466208 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466211 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466214 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466217 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466219 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466222 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466225 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466227 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466230 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466232 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.468030 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466235 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466238 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466240 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466242 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466245 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466248 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466252 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466255 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466259 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466262 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466265 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466268 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466271 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466273 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466276 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466278 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466281 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466283 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466286 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466288 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.468614 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466291 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466293 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466297 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466300 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466303 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466306 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466309 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466311 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466314 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466316 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466319 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466322 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466325 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466327 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466330 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466332 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466335 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466338 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466340 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.469100 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466343 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466346 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466351 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466354 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466357 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466362 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466365 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466368 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466370 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466373 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466375 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466378 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466381 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466384 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466386 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466390 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466393 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466396 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.469553 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466398 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466400 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466403 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466405 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466408 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466410 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466413 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466415 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466418 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466420 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466423 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466425 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466441 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466444 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466447 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466949 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466955 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466958 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466960 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466964 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.470007 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466966 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466969 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466972 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466975 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466977 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466980 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466983 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466985 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466988 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466990 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466994 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466996 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.466999 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467002 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467004 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467007 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467010 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467012 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467017 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.470515 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467020 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467024 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467026 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467029 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467032 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467034 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467037 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467039 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467042 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467045 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467047 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467050 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467052 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467055 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467057 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467060 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467062 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467064 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467067 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467069 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.470966 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467072 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467074 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467077 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467080 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467083 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467086 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467089 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467092 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467094 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467097 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467099 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467102 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467105 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467107 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467109 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467112 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467115 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467117 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467119 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467122 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.471448 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467125 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467127 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467130 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467133 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467135 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467138 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467140 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467143 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467147 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467150 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467153 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467156 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467159 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467162 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467164 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467167 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467169 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467172 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467174 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.471972 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467177 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467180 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467183 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467185 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467188 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467190 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467193 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467195 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467198 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467200 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467203 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467205 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467208 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467210 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467213 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467215 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467218 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467221 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467223 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467226 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.472421 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467229 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467231 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.467234 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.468968 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.468983 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.468989 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.468994 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.468999 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469003 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469007 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469012 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469016 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469019 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469023 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469026 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469030 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469033 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469036 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469039 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469042 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469045 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469050 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469053 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:45:18.472901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469056 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469059 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469062 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469066 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469070 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469073 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469077 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469081 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469084 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469087 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469090 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469093 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469097 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469101 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469104 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469107 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469111 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469115 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469119 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469122 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469125 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469128 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469131 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469135 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:45:18.473493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469139 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469142 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469145 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469148 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469151 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469154 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469157 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469160 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469163 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469166 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469170 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469173 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469176 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469180 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469183 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469186 2573 flags.go:64] FLAG: --help="false" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469189 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469193 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469196 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469198 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469202 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469205 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469208 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:45:18.474086 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469211 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469214 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469217 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469219 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469223 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469228 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469231 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469234 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469237 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469240 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469244 2573 flags.go:64] FLAG: --lock-file="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469246 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469249 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469252 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469271 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469274 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469277 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469280 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469284 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469288 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469292 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469295 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469300 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469303 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469307 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 18:45:18.474674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469314 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469318 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469320 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469323 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469326 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469329 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469332 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469340 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469343 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469346 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469349 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469352 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469358 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469361 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469364 2573 flags.go:64] FLAG: --port="10250" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469367 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469370 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b612e7ced5006ab4" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469373 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469376 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469379 2573 flags.go:64] FLAG: --register-node="true" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469382 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469386 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469389 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469392 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469394 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469398 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:45:18.475268 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469402 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469405 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469408 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469410 2573 flags.go:64] FLAG: --runonce="false" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469413 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469417 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469421 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469424 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469438 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469441 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469445 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469448 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469451 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469454 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469457 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469460 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469463 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469466 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469470 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469476 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469479 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469482 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469486 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469488 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469491 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:45:18.475985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469494 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469497 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469500 2573 flags.go:64] FLAG: --v="2" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469505 2573 flags.go:64] FLAG: --version="false" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469509 2573 flags.go:64] FLAG: --vmodule="" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469513 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.469517 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469688 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469693 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469696 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469699 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469703 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469707 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469710 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469713 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469716 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469719 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469722 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469724 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469727 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.476622 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469729 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469733 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469736 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469738 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469741 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469745 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469748 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469750 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469753 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469756 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469758 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469761 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469763 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469766 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469769 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469771 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469774 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469776 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469779 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469781 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.477525 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469784 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469787 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469789 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469792 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469794 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469796 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469799 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469802 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469805 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469807 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469810 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469812 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469816 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469822 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469825 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469827 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469830 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469834 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469837 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.478312 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469839 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469842 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469845 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469847 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469850 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469852 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469856 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469858 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469861 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469863 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469866 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469868 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469871 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469873 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469876 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469878 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469881 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469883 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469886 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469888 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469891 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.478837 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469893 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469896 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469899 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469902 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469904 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469908 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469910 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469913 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469916 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469919 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469922 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469925 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469927 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469930 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469932 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469935 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469937 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469940 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469942 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469945 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.479332 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469947 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469950 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469952 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469955 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469957 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469960 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469962 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469965 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469967 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469970 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469972 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469975 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.469978 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.470005 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470039 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470044 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.479833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470047 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470051 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470054 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470057 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470059 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470062 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470065 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470067 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470070 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470072 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470075 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470078 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470080 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470083 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470085 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470088 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470090 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470093 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470095 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470098 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.480218 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470101 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470103 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470106 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470108 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470111 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470113 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470116 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470120 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470123 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470127 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470130 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470133 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470136 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470140 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470143 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470145 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470147 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470150 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470152 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.480707 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470155 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470158 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470160 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470163 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470165 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470168 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470170 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470173 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470175 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470177 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470180 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470183 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470185 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470188 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470190 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470192 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470195 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470197 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470200 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470202 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.481159 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470205 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470207 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470210 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470212 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470214 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470217 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470222 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470224 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470227 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470231 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470235 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470238 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470241 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470244 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470246 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470249 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470251 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470254 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470256 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.481659 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470259 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470261 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470264 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470266 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470269 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470271 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470274 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470276 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470278 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470281 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470284 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470286 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470289 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470291 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470295 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470298 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470300 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470302 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470305 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470309 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.482108 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470312 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470315 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470318 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470320 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470323 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.470325 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.478831 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.478848 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.480136 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.480143 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480205 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480210 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480213 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480216 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480219 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480222 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480225 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480229 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480234 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480237 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.482589 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480240 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480242 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480245 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480248 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480250 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480253 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480256 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480258 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480261 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480264 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480271 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480275 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480278 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480280 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480284 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480286 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480289 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480291 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480294 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.483090 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480297 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480299 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480303 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480306 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480308 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480311 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480313 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480316 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480323 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480326 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480329 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480331 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480334 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480336 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480338 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480342 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480345 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480347 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.483661 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480350 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480352 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480355 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480358 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480361 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480364 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480366 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480369 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480372 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480374 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480377 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480379 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480382 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480384 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480387 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480390 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480393 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480395 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480398 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.484144 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480400 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480403 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480405 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480408 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480410 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480413 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480415 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480418 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480421 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480423 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480425 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480438 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480441 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480444 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480446 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480449 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480452 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480455 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480457 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.484615 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480460 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480462 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480466 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480470 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480473 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480477 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480479 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480482 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480484 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480487 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480490 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480492 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480495 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480497 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480500 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480502 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480505 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480507 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.485062 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480510 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.480539 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480682 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480686 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480689 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480692 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480695 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480698 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480700 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480703 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480706 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480709 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480712 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480714 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480717 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480720 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480723 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.485511 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480725 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480728 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480730 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480733 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480736 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480738 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480741 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480744 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480746 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480748 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480751 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480753 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480756 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480758 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480761 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480763 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480766 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480769 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480771 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480774 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.485905 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480776 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480779 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480781 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480784 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480787 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480789 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480792 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480794 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480797 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480799 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480802 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480805 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480807 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480810 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480812 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480815 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480817 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480820 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480822 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.486408 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480825 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480827 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480830 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480833 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480835 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480837 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480840 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480842 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480845 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480850 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480852 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480855 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480857 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480860 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480862 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480865 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480867 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480870 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480873 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.486877 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480878 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480880 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480883 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480885 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480888 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480890 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480893 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480896 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480899 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480901 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480905 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480908 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480911 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480914 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480916 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480919 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480922 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480924 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480927 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.487334 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480929 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480932 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480934 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480938 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480941 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480943 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480946 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480949 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480951 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480955 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480959 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480961 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480965 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.480969 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.480992 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:45:18.487833 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481024 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481029 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481032 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481034 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481037 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481040 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481042 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481045 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481048 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481051 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481053 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481056 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481060 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481064 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481066 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481069 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481071 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481074 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481077 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:45:18.488193 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481079 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481082 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481086 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481088 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481091 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481093 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481096 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481099 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481102 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481104 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481107 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481111 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481114 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481117 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481119 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481122 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481124 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481127 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481129 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481132 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:45:18.488654 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481134 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481137 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481139 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481142 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481144 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481147 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481149 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481152 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481154 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481157 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481159 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481162 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481164 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481167 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481171 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481174 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481176 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481179 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481182 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481184 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:45:18.489125 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481187 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481189 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481192 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481195 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481199 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481202 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481204 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481207 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481209 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481212 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481215 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481217 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481219 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481222 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481225 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481228 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481230 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481232 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481235 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:45:18.489644 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481237 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481240 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481243 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481246 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481248 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481251 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481253 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481257 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481260 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481262 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481265 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481268 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481270 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481273 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481275 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481278 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481282 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481284 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481287 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:45:18.490087 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481290 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481293 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481295 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481299 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481302 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481305 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481308 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481311 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:18.481313 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.481379 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.483449 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.484543 2573 server.go:1014] "Starting client certificate rotation" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.484638 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.485266 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.486586 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:45:18.490613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.487737 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:45:18.500625 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.500609 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:45:18.504533 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.504518 2573 log.go:25] "Validated CRI v1 image API" Oct 30 18:45:18.505392 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.505361 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:45:18.513245 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.513224 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 88d613c0-f86d-4437-9193-370eed9ec610:/dev/nvme0n1p3 fbc9e2df-684a-40ad-a8fa-bbaa0746e0cd:/dev/nvme0n1p4] Oct 30 18:45:18.513314 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.513242 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:45:18.520555 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.520423 2573 manager.go:217] Machine: {Timestamp:2026-10-30 18:45:18.517563983 +0000 UTC m=+0.397615108 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092706 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df66cacd9ae187d152617b2ce3a40 SystemUUID:ec2df66c-acd9-ae18-7d15-2617b2ce3a40 BootID:170d6af1-d498-47f4-a705-7768f04adc63 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d0:74:b9:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d0:74:b9:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:6d:df:18:bf:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:45:18.521074 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.521064 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:45:18.521167 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.521152 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:45:18.523939 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.523910 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:45:18.524132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.523940 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-46.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:45:18.524172 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.524135 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:45:18.524172 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.524144 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:45:18.524233 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.524190 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:45:18.524233 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.524209 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:45:18.525099 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.525089 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:45:18.528706 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.528694 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:45:18.528822 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.528812 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:45:18.529453 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.529443 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:45:18.529492 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.529461 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:45:18.529492 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.529472 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:45:18.529492 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.529481 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:45:18.529599 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.529495 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:45:18.530096 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.530075 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:45:18.534302 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.534283 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:45:18.540950 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.540929 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:45:18.542011 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.541996 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:45:18.542089 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542036 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:45:18.542241 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542228 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:45:18.542300 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542249 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:45:18.542300 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542258 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:45:18.542300 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542267 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:45:18.542300 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542277 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:45:18.542300 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542285 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:45:18.542925 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542914 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:45:18.542974 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542929 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:45:18.542974 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542940 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:45:18.542974 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542950 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:45:18.542974 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542963 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:45:18.543116 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.542977 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:45:18.545336 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.545325 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:45:18.545401 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.545341 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:45:18.549202 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.549186 2573 server.go:1263] "Started kubelet" Oct 30 18:45:18.549334 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.549303 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:45:18.549379 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.549294 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:45:18.549417 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.549387 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:45:18.549963 ip-10-0-136-46 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:45:18.551281 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.551230 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:45:18.552480 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.552219 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:45:18.556054 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.556034 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:45:18.556391 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.556373 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:45:18.557272 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.557258 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:45:18.557401 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.557377 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:18.558770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.558749 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:45:18.559311 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559291 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:45:18.559388 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559314 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:45:18.559461 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559404 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:45:18.559461 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559413 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:45:18.559753 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559738 2573 factory.go:55] Registering systemd factory Oct 30 18:45:18.559824 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.559757 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:45:18.560123 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.560108 2573 factory.go:153] Registering CRI-O factory Oct 30 18:45:18.560178 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.560128 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 18:45:18.560178 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.560169 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:45:18.560257 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.560181 2573 factory.go:103] Registering Raw factory Oct 30 18:45:18.560257 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.560198 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 18:45:18.561131 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.561118 2573 manager.go:319] Starting recovery of all containers Oct 30 18:45:18.573044 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.573030 2573 manager.go:324] Recovery completed Oct 30 18:45:18.577988 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.577970 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.579912 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.579896 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.579998 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.579930 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.579998 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.579945 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.580487 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.580474 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:45:18.580487 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.580482 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.580487 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.580485 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:45:18.580642 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.580507 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:45:18.581155 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.581130 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.581200 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.581161 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.581200 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.581175 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.582637 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.582625 2573 policy_none.go:50] "Start" Oct 30 18:45:18.582696 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.582643 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:45:18.582696 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.582657 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:45:18.584649 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.584634 2573 policy_none.go:44] "Start" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.614130 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.619646 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.619723 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.619734 2573 server.go:87] "Starting device plugin registration server" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.619945 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.619956 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.620055 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.620138 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.620145 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.620583 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.620635 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.629134 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.630291 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.630312 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.630337 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:45:18.636269 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.630369 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:45:18.684941 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.684829 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-46.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:45:18.684941 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.684838 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-46.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:45:18.685850 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.684840 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-46.us-west-2.compute.internal.18e362fdbc4f7186 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-46.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-46.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:45:18.549160326 +0000 UTC m=+0.429211453,LastTimestamp:2026-10-30 18:45:18.549160326 +0000 UTC m=+0.429211453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-46.us-west-2.compute.internal,}" Oct 30 18:45:18.720529 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.720469 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.725529 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.725512 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.725581 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.725545 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.725581 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.725562 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.725656 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.725586 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.731199 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.731181 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal"] Oct 30 18:45:18.731254 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.731238 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.732056 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.732040 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.732105 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.732071 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.732105 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.732082 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.734214 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734201 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.734369 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.734405 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734385 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.734914 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734896 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.734914 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734910 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.735040 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734928 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.735040 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734935 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.735040 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734940 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.735040 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.734946 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.737064 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.737049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.737122 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.737084 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:18.738124 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.738111 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:18.738197 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.738136 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:18.738197 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.738149 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:18.746150 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.746125 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 18:45:18.746150 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.746125 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-46.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:45:18.746306 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.746140 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-46.us-west-2.compute.internal.18e362fdbe24b119 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-46.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-46.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-46.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:45:18.579912985 +0000 UTC m=+0.459964117,LastTimestamp:2026-10-30 18:45:18.579912985 +0000 UTC m=+0.459964117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-46.us-west-2.compute.internal,}" Oct 30 18:45:18.746498 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.746479 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 18:45:18.750212 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.750195 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:45:18.750287 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.750244 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j2hvp" Oct 30 18:45:18.760411 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.760385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.760411 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.760410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.760537 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.760427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.781154 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.781137 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.783186 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.783169 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.795528 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.795509 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 18:45:18.848359 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.848328 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.848359 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.848359 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-46.us-west-2.compute.internal\": node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:18.861187 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.861252 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861219 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.861252 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.861356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.861356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861307 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.861356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:18.861326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:18.956950 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:18.956923 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:19.057250 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:19.057169 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:19.084509 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:19.084488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:19.087575 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:19.087557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:19.157749 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:19.157560 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:20.890063 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:20.890035 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:20.990183 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:20.990110 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.091030 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.091000 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.115760 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.115742 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:45:21.116171 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.115876 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a08f964132a324d829023bb7d5fff707-ee203c489136d9c1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-46.us-west-2.compute.internal.18e362fdbe250ebe\": read tcp 10.0.136.46:33344->18.224.117.112:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-46.us-west-2.compute.internal.18e362fdbe250ebe default 4770 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-46.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-46.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-46.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:45:18 +0000 UTC,LastTimestamp:2026-10-30 18:45:18.732076233 +0000 UTC m=+0.612127360,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-46.us-west-2.compute.internal,}" Oct 30 18:45:21.191064 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.191044 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.291341 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.291273 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.338881 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.338848 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:45:21.339863 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.339844 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:45:21.339968 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.339881 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:45:21.339968 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.339896 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:45:21.343685 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.343669 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:45:21.343769 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.343755 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:45:21.391646 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.391622 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.392719 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.392693 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:40:20 +0000 UTC" deadline="2028-07-09 12:24:31.15221774 +0000 UTC" Oct 30 18:45:21.392771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.392719 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14825h39m9.759500993s" Oct 30 18:45:21.492002 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.491973 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.593140 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.593063 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.694009 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.693972 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.759558 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.759531 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:45:21.790546 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:21.790515 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dce357bba57b17e7ff4a65afbed63d.slice/crio-34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d WatchSource:0}: Error finding container 34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d: Status 404 returned error can't find the container with id 34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d Oct 30 18:45:21.792112 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:21.792086 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode611e756739ad427bda17a40f0e6a03c.slice/crio-16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3 WatchSource:0}: Error finding container 16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3: Status 404 returned error can't find the container with id 16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3 Oct 30 18:45:21.794242 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:21.794223 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:45:21.797065 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.797048 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:45:21.806704 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.806687 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:45:21.889843 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.889771 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:21.950910 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.950882 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:45:21.951932 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:21.951914 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:45:22.008022 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.007996 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:45:22.162744 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.162675 2573 apiserver.go:52] "Watching apiserver" Oct 30 18:45:22.188866 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.188843 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:45:22.261960 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.261931 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:45:22.262241 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.262225 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jgml5","kube-system/global-pull-secret-syncer-vnc44","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-lbk7g","openshift-network-operator/iptables-alerter-zsxjb","openshift-ovn-kubernetes/ovnkube-node-djnd7","kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-5xbl5","openshift-image-registry/node-ca-4cm6p","kube-system/konnectivity-agent-rs2ft","openshift-dns/node-resolver-s65mr","openshift-multus/multus-additional-cni-plugins-qlzjf","openshift-multus/multus-pskdl"] Oct 30 18:45:22.266015 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.265990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.268199 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.268181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.270398 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.270382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.272673 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.272598 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.274720 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.274702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:22.274814 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.274789 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:22.276847 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.276834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.278955 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.278942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.281243 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.281229 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.283550 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.283532 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.283650 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.283605 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:22.285680 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.285666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.285734 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.285723 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:22.287923 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.287907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.290931 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.290909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.293604 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.293563 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerStarted","Data":"16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3"} Oct 30 18:45:22.293696 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.293610 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"29dce357bba57b17e7ff4a65afbed63d","Type":"ContainerStarted","Data":"34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d"} Oct 30 18:45:22.301151 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301130 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.301151 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcm7\" (UniqueName: \"kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.301326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301317 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301375 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301430 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljzq\" (UniqueName: \"kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301605 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdln\" (UniqueName: \"kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2bv\" (UniqueName: \"kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.301993 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.301989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6px\" (UniqueName: \"kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302282 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.302708 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd89\" (UniqueName: \"kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnhg\" (UniqueName: \"kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302425 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302489 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302511 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302531 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:22.303506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302654 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgpj\" (UniqueName: \"kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw889\" (UniqueName: \"kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.304282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.302943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.316434 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.316413 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:45:22.316514 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.316416 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.316585 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.316569 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.316585 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.316580 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n4vwp\"" Oct 30 18:45:22.317997 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.317983 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:45:22.318427 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.318414 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.318535 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.318522 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:45:22.318597 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.318585 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpjwm\"" Oct 30 18:45:22.318639 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.318631 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:45:22.318765 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.318751 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.320511 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.320493 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:45:22.320583 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.320500 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.320583 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.320545 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.320583 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.320552 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lj6ts\"" Oct 30 18:45:22.322575 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.322560 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:45:22.322757 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.322743 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56wtl\"" Oct 30 18:45:22.328201 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328179 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.328297 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328270 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.328576 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328561 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:45:22.328649 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328634 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.328649 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328636 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77vpl\"" Oct 30 18:45:22.328829 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.328817 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5tsjg\"" Oct 30 18:45:22.359423 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.359392 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:45:22.359423 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.359421 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.359661 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.359646 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:45:22.359699 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.359666 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.364335 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.364320 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:45:22.364415 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.364361 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:45:22.365119 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365105 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:22.365528 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365513 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz6tx\"" Oct 30 18:45:22.365576 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365549 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r47g8\"" Oct 30 18:45:22.365880 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365867 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:45:22.365943 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365893 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29qlm\"" Oct 30 18:45:22.365943 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.365909 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.370121 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.370107 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:45:22.370229 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.370161 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:45:22.370229 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.370171 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:45:22.370229 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.370204 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlzfm" Oct 30 18:45:22.391904 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.391880 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:45:22.403518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.403621 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.403621 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.403621 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403643 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.403716 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6px\" (UniqueName: \"kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.403771 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.403778 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:22.903757761 +0000 UTC m=+3.152030938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403864 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.403937 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.403983 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:22.903969363 +0000 UTC m=+3.152242543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.403979 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.404059 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404071 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404116 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404143 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404144 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404194 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404265 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.404657 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404421 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd89\" (UniqueName: \"kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404456 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnhg\" (UniqueName: \"kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404789 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.405470 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404818 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404856 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.404986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405097 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405171 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405198 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgpj\" (UniqueName: \"kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405329 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw889\" (UniqueName: \"kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcm7\" (UniqueName: \"kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.406963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.405932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406389 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljzq\" (UniqueName: \"kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.406929 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407016 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407119 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.407859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407239 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407564 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407644 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.407944 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408028 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408454 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408461 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdln\" (UniqueName: \"kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.408613 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408618 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408647 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408846 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408916 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.408967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409003 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409057 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2bv\" (UniqueName: \"kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409114 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.409449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409250 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.410184 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409249 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.410184 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409313 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.410184 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409468 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.410184 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.410184 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.409710 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.410674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.410652 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.411505 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.411484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.411651 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.411628 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.411749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.411730 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.460879 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.460820 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:22.460879 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.460845 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:22.460879 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.460860 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:22.461043 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.460930 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:22.960912135 +0000 UTC m=+3.209185326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:22.462979 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.462946 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd89\" (UniqueName: \"kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.463069 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.463001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnhg\" (UniqueName: \"kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.463069 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.463021 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6px\" (UniqueName: \"kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.464665 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.464646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw889\" (UniqueName: \"kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.466240 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.466196 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcm7\" (UniqueName: \"kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.466343 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.466318 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgpj\" (UniqueName: \"kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.467324 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.467303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdln\" (UniqueName: \"kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.467626 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.467611 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2bv\" (UniqueName: \"kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.467667 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.467656 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljzq\" (UniqueName: \"kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.578530 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.578504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:45:22.587950 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.587922 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87d2cac_3415_4c41_9ba2_de3dd8ee236f.slice/crio-9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7 WatchSource:0}: Error finding container 9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7: Status 404 returned error can't find the container with id 9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7 Oct 30 18:45:22.599535 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.599518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:45:22.602247 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.602111 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:45:22.603195 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.603180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pskdl" Oct 30 18:45:22.605188 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.605166 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:45:22.609505 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.609445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:45:22.609680 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.609658 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cee29d4_8788_4842_b8c5_a2f6e6f3996a.slice/crio-2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f WatchSource:0}: Error finding container 2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f: Status 404 returned error can't find the container with id 2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f Oct 30 18:45:22.612441 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.612407 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f076db4_f03f_4bf3_be39_e899370f953c.slice/crio-e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc WatchSource:0}: Error finding container e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc: Status 404 returned error can't find the container with id e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc Oct 30 18:45:22.616111 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.616077 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb8307e_d525_404e_ac04_ec75650aa4f2.slice/crio-ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1 WatchSource:0}: Error finding container ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1: Status 404 returned error can't find the container with id ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1 Oct 30 18:45:22.619543 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.619526 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e48c7c_d60b_4daf_aa1a_b4578e3d910f.slice/crio-46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35 WatchSource:0}: Error finding container 46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35: Status 404 returned error can't find the container with id 46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35 Oct 30 18:45:22.621993 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.621968 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf3c0f3_1cf9_4fc3_8973_ead2c2dc36a6.slice/crio-f3d9bc637a6b6df0f8b68cce1ef9dee3f7b3868dd9e231e26a96039af46b547f WatchSource:0}: Error finding container f3d9bc637a6b6df0f8b68cce1ef9dee3f7b3868dd9e231e26a96039af46b547f: Status 404 returned error can't find the container with id f3d9bc637a6b6df0f8b68cce1ef9dee3f7b3868dd9e231e26a96039af46b547f Oct 30 18:45:22.627050 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.627034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:22.630156 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.630141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s65mr" Oct 30 18:45:22.634101 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.634034 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:22.636803 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.636784 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150b998b_9608_4846_ac54_8a8ec8a7c620.slice/crio-47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb WatchSource:0}: Error finding container 47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb: Status 404 returned error can't find the container with id 47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb Oct 30 18:45:22.640681 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.640663 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3eacfd_ea78_48c0_8968_da01c34ec561.slice/crio-cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e WatchSource:0}: Error finding container cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e: Status 404 returned error can't find the container with id cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e Oct 30 18:45:22.643562 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:22.643545 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde998501_b81d_4fa5_b100_34ee068f8b03.slice/crio-572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31 WatchSource:0}: Error finding container 572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31: Status 404 returned error can't find the container with id 572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31 Oct 30 18:45:22.913361 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.913293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:22.913361 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:22.913331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:22.913526 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.913438 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:22.913526 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.913487 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:23.913474567 +0000 UTC m=+4.161747731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:22.913597 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.913537 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:22.913597 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:22.913566 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:23.91355626 +0000 UTC m=+4.161829425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:23.014222 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.014174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:23.014386 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.014372 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:23.014435 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.014393 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:23.014435 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.014405 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:23.014517 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.014458 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:24.01444014 +0000 UTC m=+4.262713311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:23.271424 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.271346 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" event={"ID":"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6","Type":"ContainerStarted","Data":"f3d9bc637a6b6df0f8b68cce1ef9dee3f7b3868dd9e231e26a96039af46b547f"} Oct 30 18:45:23.272378 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.272349 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pskdl" event={"ID":"acb8307e-d525-404e-ac04-ec75650aa4f2","Type":"ContainerStarted","Data":"ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1"} Oct 30 18:45:23.273498 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.273476 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cm6p" event={"ID":"c87d2cac-3415-4c41-9ba2-de3dd8ee236f","Type":"ContainerStarted","Data":"9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7"} Oct 30 18:45:23.274615 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.274597 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsxjb" event={"ID":"38e48c7c-d60b-4daf-aa1a-b4578e3d910f","Type":"ContainerStarted","Data":"46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35"} Oct 30 18:45:23.275680 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.275661 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" event={"ID":"2f076db4-f03f-4bf3-be39-e899370f953c","Type":"ContainerStarted","Data":"e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc"} Oct 30 18:45:23.277236 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.277183 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerStarted","Data":"2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f"} Oct 30 18:45:23.278580 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.278558 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31"} Oct 30 18:45:23.281518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.281494 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s65mr" event={"ID":"0d3eacfd-ea78-48c0-8968-da01c34ec561","Type":"ContainerStarted","Data":"cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e"} Oct 30 18:45:23.282581 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.282561 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rs2ft" event={"ID":"150b998b-9608-4846-ac54-8a8ec8a7c620","Type":"ContainerStarted","Data":"47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb"} Oct 30 18:45:23.372976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.372927 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:40:22 +0000 UTC" deadline="2028-04-08 21:16:31.789582588 +0000 UTC" Oct 30 18:45:23.372976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.372972 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12626h31m8.416614485s" Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.920338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:23.920386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.920514 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.920583 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:25.920565229 +0000 UTC m=+6.168838393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.920647 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:23.920728 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:23.920682 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:25.920670966 +0000 UTC m=+6.168944136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:24.021720 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.021375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:24.021720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.021575 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:24.021720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.021594 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:24.021720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.021606 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:24.021720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.021663 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:26.021645877 +0000 UTC m=+6.269919046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.265232 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.265379 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.265468 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.265545 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.265602 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:24.265752 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:24.265669 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:24.373400 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.373255 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:40:22 +0000 UTC" deadline="2028-06-15 00:00:19.531538582 +0000 UTC" Oct 30 18:45:24.373400 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:24.373288 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14237h14m55.158254723s" Oct 30 18:45:25.306509 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:25.305362 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"29dce357bba57b17e7ff4a65afbed63d","Type":"ContainerStarted","Data":"f48df332f7dffb05e74511fb5932ff55269f607f721d8e4f7462bceb9482f792"} Oct 30 18:45:25.937243 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:25.937192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:25.937754 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:25.937264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:25.937754 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:25.937403 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:25.937754 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:25.937459 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:29.937442462 +0000 UTC m=+10.185715643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:25.937754 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:25.937522 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:25.937754 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:25.937555 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:29.937543334 +0000 UTC m=+10.185816503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:26.037960 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.037926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:26.038104 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.038085 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:26.038170 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.038113 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:26.038170 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.038126 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:26.038388 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.038180 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:30.038163712 +0000 UTC m=+10.286436892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:26.265217 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.265132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:26.265854 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.265824 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:26.265963 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.265910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:26.266021 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.265999 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:26.266072 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.266045 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:26.266135 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:26.266115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:26.309894 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.309859 2573 generic.go:356] "Generic (PLEG): container finished" podID="e611e756739ad427bda17a40f0e6a03c" containerID="a266f0efa4b741781d52d922ed01bb177f2f913fd8f852cf3a29fa045e4e37d5" exitCode=0 Oct 30 18:45:26.310007 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.309949 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerDied","Data":"a266f0efa4b741781d52d922ed01bb177f2f913fd8f852cf3a29fa045e4e37d5"} Oct 30 18:45:26.313500 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.313467 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="46027b51264cd72c50a9ba2ebe28a78429cdaad053967156a35b9374a3a693de" exitCode=0 Oct 30 18:45:26.314119 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.314060 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"46027b51264cd72c50a9ba2ebe28a78429cdaad053967156a35b9374a3a693de"} Oct 30 18:45:26.423790 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:26.422551 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" podStartSLOduration=5.422533326 podStartE2EDuration="5.422533326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:25.429117129 +0000 UTC m=+5.677390315" watchObservedRunningTime="2026-10-30 18:45:26.422533326 +0000 UTC m=+6.670806513" Oct 30 18:45:27.322521 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:27.321816 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rs2ft" event={"ID":"150b998b-9608-4846-ac54-8a8ec8a7c620","Type":"ContainerStarted","Data":"1c565428a7659c334fb99e1eb277f1e60827b32605550cf06fdecfac7c31f601"} Oct 30 18:45:27.335875 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:27.335341 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerStarted","Data":"3e01f554dedd40dc520368dcb044899abbfd0f237560391b9ca31b7815488c49"} Oct 30 18:45:27.546200 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:27.544927 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rs2ft" podStartSLOduration=3.325376295 podStartE2EDuration="7.544908308s" totalImagesPullingTime="4.219532013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.43393638 +0000 UTC m=+7.682209564" watchObservedRunningTime="2026-10-30 18:45:27.544908308 +0000 UTC m=+7.793181492" Oct 30 18:45:28.265477 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:28.265404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:28.265655 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:28.265544 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:28.265655 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:28.265630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:28.265767 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:28.265721 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:28.265819 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:28.265766 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:28.265854 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:28.265838 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:28.342936 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:28.342902 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4cm6p" event={"ID":"c87d2cac-3415-4c41-9ba2-de3dd8ee236f","Type":"ContainerStarted","Data":"3aeda1e4a72d4d445a7b874e1292bc1c8bc194a6fffef3464c7653bd57839102"} Oct 30 18:45:28.457109 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:28.457055 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" podStartSLOduration=7.457036007 podStartE2EDuration="7.457036007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.546175851 +0000 UTC m=+7.794449039" watchObservedRunningTime="2026-10-30 18:45:28.457036007 +0000 UTC m=+8.705309194" Oct 30 18:45:29.347329 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:29.347282 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" event={"ID":"2f076db4-f03f-4bf3-be39-e899370f953c","Type":"ContainerStarted","Data":"4ba7e6bc525db798e99de0e17883670b615771211e853c6897e355dde47af64f"} Oct 30 18:45:29.971805 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:29.971774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:29.971805 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:29.971812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:29.972041 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:29.971916 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:29.972041 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:29.971963 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:37.971945571 +0000 UTC m=+18.220218748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:29.972366 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:29.972286 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:29.972366 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:29.972339 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:37.972325318 +0000 UTC m=+18.220598498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:30.072469 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:30.072171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:30.072469 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.072363 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:30.072469 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.072383 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:30.072469 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.072394 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:30.072469 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.072438 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:38.072425495 +0000 UTC m=+18.320698672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:30.265310 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:30.265165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:30.265451 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:30.265362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:30.265517 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.265454 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:30.265638 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:30.265594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:30.265802 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.265709 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:30.265802 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:30.265757 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:30.913888 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:30.913863 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:45:31.260886 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.260788 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:45:30.913885726Z","UUID":"b4c6610f-63f5-41eb-84eb-257d397dcdab","Handler":null,"Name":"","Endpoint":""} Oct 30 18:45:31.262191 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.262166 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:45:31.262293 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.262199 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:45:31.354770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.354635 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" event={"ID":"2f076db4-f03f-4bf3-be39-e899370f953c","Type":"ContainerStarted","Data":"bcf9df1b072384974747cdc44e06a9b82fbad1e18da0b2d81728e6cb25729ca4"} Oct 30 18:45:31.357392 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.357323 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerStarted","Data":"0825d0db7e306c4df27214ed68643bd5829b881f6fbec70b2b79c8e04e0720cf"} Oct 30 18:45:31.359477 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.359402 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s65mr" event={"ID":"0d3eacfd-ea78-48c0-8968-da01c34ec561","Type":"ContainerStarted","Data":"5130ad7dd43d35de35f167379f75912fca8bbf171623e9134e2c28967940695b"} Oct 30 18:45:31.361279 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.361254 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pskdl" event={"ID":"acb8307e-d525-404e-ac04-ec75650aa4f2","Type":"ContainerStarted","Data":"227bf5e30c880c30cd42a2b0b3b0e656e3c82ae39b4b15eaf56ae7d9e2d12a79"} Oct 30 18:45:31.493097 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.492891 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4cm6p" podStartSLOduration=6.248558529 podStartE2EDuration="11.492872758s" totalImagesPullingTime="5.244314229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:28.457727213 +0000 UTC m=+8.706000400" watchObservedRunningTime="2026-10-30 18:45:31.492872758 +0000 UTC m=+11.741145943" Oct 30 18:45:31.608987 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.608924 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s65mr" podStartSLOduration=3.270072646 podStartE2EDuration="11.608906026s" totalImagesPullingTime="8.33883338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:31.608328198 +0000 UTC m=+11.856601384" watchObservedRunningTime="2026-10-30 18:45:31.608906026 +0000 UTC m=+11.857179215" Oct 30 18:45:31.733796 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.733750 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pskdl" podStartSLOduration=3.89831255 podStartE2EDuration="11.733735964s" totalImagesPullingTime="7.835423414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:31.733533222 +0000 UTC m=+11.981806443" watchObservedRunningTime="2026-10-30 18:45:31.733735964 +0000 UTC m=+11.982009157" Oct 30 18:45:31.880362 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.880334 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:31.881168 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.881149 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:31.945969 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.945945 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:31.946413 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:31.946393 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:32.265161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:32.265283 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:32.265658 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:32.265742 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:32.265800 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:32.265936 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:32.265884 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:32.365946 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:32.365919 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="0825d0db7e306c4df27214ed68643bd5829b881f6fbec70b2b79c8e04e0720cf" exitCode=0 Oct 30 18:45:32.366058 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:32.366033 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"0825d0db7e306c4df27214ed68643bd5829b881f6fbec70b2b79c8e04e0720cf"} Oct 30 18:45:33.369080 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.369038 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" event={"ID":"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6","Type":"ContainerStarted","Data":"fe3c7668429dce52c7606a37fba0cc5730f1a6da991e17ff5a3854a3d5803bbe"} Oct 30 18:45:33.370910 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.370886 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zsxjb" event={"ID":"38e48c7c-d60b-4daf-aa1a-b4578e3d910f","Type":"ContainerStarted","Data":"dba6b1d8f632f19064fe8237682f1b25f61504bd0c7e72d30c2cda78e307462b"} Oct 30 18:45:33.372769 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.372746 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" event={"ID":"2f076db4-f03f-4bf3-be39-e899370f953c","Type":"ContainerStarted","Data":"99edeee538c5f8dc3a63b90ace550be02a3ed62d3c32f50c9f185992b4a932c8"} Oct 30 18:45:33.485000 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.484955 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" podStartSLOduration=3.3940252920000002 podStartE2EDuration="13.484942077s" totalImagesPullingTime="10.090916785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:33.484661956 +0000 UTC m=+13.732935146" watchObservedRunningTime="2026-10-30 18:45:33.484942077 +0000 UTC m=+13.733215262" Oct 30 18:45:33.604318 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.604278 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" podStartSLOduration=4.23224448 podStartE2EDuration="13.604255834s" totalImagesPullingTime="9.372011354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:33.603777505 +0000 UTC m=+13.852050691" watchObservedRunningTime="2026-10-30 18:45:33.604255834 +0000 UTC m=+13.852529020" Oct 30 18:45:33.715281 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:33.715243 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zsxjb" podStartSLOduration=5.203226102 podStartE2EDuration="13.715230016s" totalImagesPullingTime="8.512003914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:33.714960136 +0000 UTC m=+13.963233322" watchObservedRunningTime="2026-10-30 18:45:33.715230016 +0000 UTC m=+13.963503199" Oct 30 18:45:34.263110 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:34.263085 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:34.263276 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:34.263149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:34.263276 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:34.263161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:34.263395 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:34.263286 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:34.263395 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:34.263353 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:34.263496 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:34.263457 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:34.376515 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:34.376467 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="8c10ab35d7d168719fee3535e6ca18768883bc557f7ed0b41752d70ad87132a4" exitCode=0 Oct 30 18:45:34.377020 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:34.376573 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"8c10ab35d7d168719fee3535e6ca18768883bc557f7ed0b41752d70ad87132a4"} Oct 30 18:45:36.263587 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:36.262643 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:36.263587 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:36.262966 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:36.263587 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:36.263368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:36.263587 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:36.263463 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:36.264265 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:36.263606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:36.264265 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:36.263732 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:36.383656 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:36.383624 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="78e723517c5e25b3b8783c4f218427684b10f0273253b1939eaae40118305fdf" exitCode=0 Oct 30 18:45:36.383815 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:36.383677 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"78e723517c5e25b3b8783c4f218427684b10f0273253b1939eaae40118305fdf"} Oct 30 18:45:38.036238 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.036195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:38.036743 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.036249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:38.036743 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.036350 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:38.036743 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.036409 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret podName:9cdd95a5-32ae-4c7a-a067-bc40ad4bc826 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.036393265 +0000 UTC m=+34.284666433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret") pod "global-pull-secret-syncer-vnc44" (UID: "9cdd95a5-32ae-4c7a-a067-bc40ad4bc826") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:45:38.036743 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.036350 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:38.036743 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.036456 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.036446401 +0000 UTC m=+34.284719565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:38.137473 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.137440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:38.137620 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.137584 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:45:38.137620 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.137602 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:45:38.137620 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.137614 2573 projected.go:196] Error preparing data for projected volume kube-api-access-g2gll for pod openshift-network-diagnostics/network-check-target-lbk7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:38.137753 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.137674 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll podName:bb7ee20f-a7f2-4994-abea-4c233d8b8da3 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.137656192 +0000 UTC m=+34.385929371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2gll" (UniqueName: "kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll") pod "network-check-target-lbk7g" (UID: "bb7ee20f-a7f2-4994-abea-4c233d8b8da3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.262356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.262391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:38.262356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.262482 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.262559 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:38.262781 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:38.262611 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:39.391976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.391692 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"12f9590d3934adbc02797dab9fac2baa2739a63cca55e057cb2d275d8b33226b"} Oct 30 18:45:39.392386 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.392021 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"95f61a8d1f810b3b741254e0cb742f7b556d31d62d6337926427b6200079d588"} Oct 30 18:45:39.392386 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.392083 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"5faeb5f2811245c535c28fe87978f9436b1dd3f6d4fb3abbca3d23279281f2ea"} Oct 30 18:45:39.392386 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.392130 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"49e5c956ed0306d2093c020a962f01cfd9ed42b8917807ea5572580465ec4c92"} Oct 30 18:45:39.392386 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.392226 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"286167ad5906a8185452c3c1acd44206fd8e4ca7fcacfebcc0a751e268e2f35b"} Oct 30 18:45:39.392386 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.392266 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"80fc456c00e135ca67b465110cbc4157dc6c6d4c0e4c03941e189c83baf59535"} Oct 30 18:45:39.394740 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.394715 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="8e56309d3ec2c3a24ca0cf327f9de01d0eed76f22054a9b3f8024f958ca38389" exitCode=0 Oct 30 18:45:39.394841 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:39.394750 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"8e56309d3ec2c3a24ca0cf327f9de01d0eed76f22054a9b3f8024f958ca38389"} Oct 30 18:45:40.264131 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:40.264094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:40.264302 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:40.264261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:40.264302 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:40.264278 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:40.264436 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:40.264364 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:40.264495 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:40.264399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:40.264562 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:40.264539 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:40.399945 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:40.399913 2573 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="52d537218be09499b353d61b82b34082050f82b480f588dfe50b33e916ca4f5b" exitCode=0 Oct 30 18:45:40.400417 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:40.399966 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerDied","Data":"52d537218be09499b353d61b82b34082050f82b480f588dfe50b33e916ca4f5b"} Oct 30 18:45:41.405866 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:41.405823 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" event={"ID":"7cee29d4-8788-4842-b8c5-a2f6e6f3996a","Type":"ContainerStarted","Data":"316a79f12aef3ab31e06b4f385d18ab221ba462e01d9d84977b915c2f9b40be3"} Oct 30 18:45:41.408758 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:41.408735 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"2b818c020d5c887dbc50f95b6d96015cd649ddefa2f7fd1977468e0fbdb86ed1"} Oct 30 18:45:41.523830 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:41.523758 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qlzjf" podStartSLOduration=8.553589275 podStartE2EDuration="21.523745651s" totalImagesPullingTime="12.970156376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:41.523624169 +0000 UTC m=+21.771897358" watchObservedRunningTime="2026-10-30 18:45:41.523745651 +0000 UTC m=+21.772018836" Oct 30 18:45:42.262602 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:42.262572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:42.262602 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:42.262592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:42.262868 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:42.262674 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:42.262868 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:42.262711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:42.262868 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:42.262769 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:42.262868 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:42.262853 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:43.417004 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:43.416976 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" event={"ID":"de998501-b81d-4fa5-b100-34ee068f8b03","Type":"ContainerStarted","Data":"a556d53d9d00ed2695f8072d7091acb5b034b5d467c96b3cad12ab79e24fe8e5"} Oct 30 18:45:43.417726 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:43.417238 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:43.432054 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:43.432033 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:43.683835 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:43.683749 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" podStartSLOduration=7.5338796089999995 podStartE2EDuration="23.683736743s" totalImagesPullingTime="16.149857134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:43.683511536 +0000 UTC m=+23.931784721" watchObservedRunningTime="2026-10-30 18:45:43.683736743 +0000 UTC m=+23.932009958" Oct 30 18:45:44.262773 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.262748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:44.262773 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.262753 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:44.262948 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:44.262873 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:44.262948 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.262917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:44.263016 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:44.263004 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:44.263113 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:44.263074 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:44.419490 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.419468 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:44.419808 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.419526 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:44.433383 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:44.433366 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:45:45.884558 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.884523 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnc44"] Oct 30 18:45:45.885026 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.884629 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:45.885026 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:45.884713 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:45.885150 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.885065 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lbk7g"] Oct 30 18:45:45.885150 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.885140 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:45.885252 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:45.885221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:45.931725 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.931701 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgml5"] Oct 30 18:45:45.931820 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:45.931783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:45.931875 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:45.931856 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:47.263043 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:47.263007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:47.263469 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:47.263153 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:48.263087 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:48.263062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:48.263087 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:48.263081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:48.263611 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:48.263162 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:48.263611 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:48.263281 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:49.262890 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:49.262860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:49.263042 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:49.262973 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:50.263155 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:50.263119 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:50.263650 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:50.263244 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:50.263650 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:50.263255 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:50.263650 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:50.263333 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:51.263355 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:51.263328 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:51.263749 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:51.263429 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnc44" podUID="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" Oct 30 18:45:52.262382 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.262353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:52.262502 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.262473 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lbk7g" podUID="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" Oct 30 18:45:52.262560 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.262517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:52.262619 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.262603 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jgml5" podUID="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" Oct 30 18:45:52.489056 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.489003 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeReady" Oct 30 18:45:52.489392 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.489098 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:45:52.601431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.601407 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-54zh4"] Oct 30 18:45:52.661105 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.661080 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jnbnw"] Oct 30 18:45:52.661241 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.661221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:45:52.674037 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.674020 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzdtx"] Oct 30 18:45:52.674153 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.674136 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.686562 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.686540 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fj8mt"] Oct 30 18:45:52.686661 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.686646 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:45:52.708850 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.708828 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht"] Oct 30 18:45:52.709515 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.708905 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:45:52.713471 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.713454 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.732330 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.732307 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:45:52.732422 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.732412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.746388 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.746349 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc"] Oct 30 18:45:52.746486 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.746470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.747845 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.747821 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 18:45:52.747931 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.747917 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dfwxh\"" Oct 30 18:45:52.748766 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.748744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99zj\" (UniqueName: \"kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj\") pod \"kas-connection-checker-685895c7c9-54zh4\" (UID: \"497867c2-a03d-4717-b845-1d0ef71495d5\") " pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:45:52.770037 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.770018 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5978b965c7-kbthb"] Oct 30 18:45:52.770241 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.770147 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.780674 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.780656 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nlzxq"] Oct 30 18:45:52.780788 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.780772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.803186 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.803167 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m"] Oct 30 18:45:52.803298 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.803278 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:52.813510 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.813492 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.813598 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.813549 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.813673 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.813558 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 18:45:52.813673 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.813554 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.814319 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.814305 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 18:45:52.818267 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.818251 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx"] Oct 30 18:45:52.818368 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.818353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:52.840796 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.840777 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k"] Oct 30 18:45:52.840921 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.840910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:52.849374 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5p78\" (UniqueName: \"kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.849518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849388 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.849518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkv2k\" (UniqueName: \"kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k\") pod \"kas-connection-checker-685895c7c9-bzdtx\" (UID: \"41bb2e81-6a7b-434e-a6c4-184beef52d40\") " pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:45:52.849518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.849640 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849640 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849701 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849734 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849764 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849797 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.849826 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849855 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.849855 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.849910 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849856 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.849910 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.849969 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jh9\" (UniqueName: \"kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.849969 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.849925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxh9\" (UniqueName: \"kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9\") pod \"kas-connection-checker-685895c7c9-fj8mt\" (UID: \"8c417afb-ca2e-4706-8004-f6651dd11f16\") " pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:45:52.850032 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.850010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99zj\" (UniqueName: \"kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj\") pod \"kas-connection-checker-685895c7c9-54zh4\" (UID: \"497867c2-a03d-4717-b845-1d0ef71495d5\") " pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:45:52.857004 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.856989 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-95sk7"] Oct 30 18:45:52.857096 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.857083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:45:52.861805 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861778 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:45:52.861903 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861839 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 18:45:52.861903 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861874 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 18:45:52.861903 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861890 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:45:52.861903 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861839 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 18:45:52.862081 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861931 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 18:45:52.862081 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861944 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 18:45:52.862081 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861964 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.862081 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861972 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:45:52.862081 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.861994 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 18:45:52.862298 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862097 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.862298 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862166 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r2fz8\"" Oct 30 18:45:52.862298 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862248 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:45:52.862298 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862282 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6fg8\"" Oct 30 18:45:52.862490 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862300 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 18:45:52.862490 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862370 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hbsvt\"" Oct 30 18:45:52.862490 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862427 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kzsm2\"" Oct 30 18:45:52.862490 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.862470 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ffnnw\"" Oct 30 18:45:52.868896 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.868882 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 18:45:52.869156 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.869142 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-68fw6\"" Oct 30 18:45:52.878169 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.878154 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:45:52.878288 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.878274 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:52.892381 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.892362 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw"] Oct 30 18:45:52.892478 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.892465 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:52.904761 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.904745 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xw9bs"] Oct 30 18:45:52.904874 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.904860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:45:52.910104 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.909971 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99zj\" (UniqueName: \"kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj\") pod \"kas-connection-checker-685895c7c9-54zh4\" (UID: \"497867c2-a03d-4717-b845-1d0ef71495d5\") " pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:45:52.914117 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914097 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.914434 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914412 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.914529 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914420 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.914529 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914450 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.914616 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914567 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 18:45:52.914616 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914593 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.914616 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914603 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.914616 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914613 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.914868 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914853 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.914924 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914867 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 18:45:52.914924 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914884 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:45:52.914924 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914860 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 18:45:52.914924 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.914870 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 18:45:52.915196 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.915183 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 18:45:52.915688 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.915675 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 18:45:52.934349 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.934333 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q7r4n"] Oct 30 18:45:52.934457 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.934443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:52.948586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948571 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-54zh4"] Oct 30 18:45:52.948664 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948595 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzdtx"] Oct 30 18:45:52.948664 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948608 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jnbnw"] Oct 30 18:45:52.948664 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948621 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht"] Oct 30 18:45:52.948664 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948636 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m"] Oct 30 18:45:52.948664 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948649 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc"] Oct 30 18:45:52.948850 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948663 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:52.948850 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948666 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:45:52.948850 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948810 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k"] Oct 30 18:45:52.948850 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948837 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-95sk7"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948861 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xw9bs"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948881 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fj8mt"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948899 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948929 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948948 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5978b965c7-kbthb"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948962 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx"] Oct 30 18:45:52.948976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948978 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7r4n"] Oct 30 18:45:52.949230 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.948994 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nlzxq"] Oct 30 18:45:52.949550 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.949533 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 18:45:52.949637 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.949553 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-smsmz\"" Oct 30 18:45:52.949637 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.949557 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tqgpw\"" Oct 30 18:45:52.949637 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.949576 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 18:45:52.949752 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.949713 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q6v4b\"" Oct 30 18:45:52.950702 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:52.950770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.950770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:52.950834 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.950834 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.950834 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxh9\" (UniqueName: \"kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9\") pod \"kas-connection-checker-685895c7c9-fj8mt\" (UID: \"8c417afb-ca2e-4706-8004-f6651dd11f16\") " pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:45:52.950915 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.950975 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.950956 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:45:52.951060 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.950981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.951060 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmlw\" (UniqueName: \"kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:52.951060 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.951022 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.451004414 +0000 UTC m=+33.699277587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:45:52.951060 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.951289 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jh9\" (UniqueName: \"kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.951289 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.951141 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:45:52.951289 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.951185 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.451170331 +0000 UTC m=+33.699443513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:45:52.951289 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.951289 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951352 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89cg\" (UniqueName: \"kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkv2k\" (UniqueName: \"kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k\") pod \"kas-connection-checker-685895c7c9-bzdtx\" (UID: \"41bb2e81-6a7b-434e-a6c4-184beef52d40\") " pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvt7\" (UniqueName: \"kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7\") pod \"volume-data-source-validator-58c9f5f46d-2tt7k\" (UID: \"7224a54e-ce8c-4b82-9247-11376266019b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:45:52.951574 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q5j\" (UniqueName: \"kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951770 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4tm\" (UniqueName: \"kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.951897 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5p78\" (UniqueName: \"kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951959 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.951972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.952252 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.952279 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.952300 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.452285357 +0000 UTC m=+33.700558522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.952291 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:52.952347 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.452337668 +0000 UTC m=+33.700610833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:45:52.952373 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:52.952825 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.952825 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952471 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:52.952886 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.952842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.953280 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.953264 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.954454 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.954439 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:52.956482 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.956468 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pwfgl\"" Oct 30 18:45:52.982338 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:52.982316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:45:53.010479 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.010459 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxh9\" (UniqueName: \"kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9\") pod \"kas-connection-checker-685895c7c9-fj8mt\" (UID: \"8c417afb-ca2e-4706-8004-f6651dd11f16\") " pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:45:53.010562 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.010520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:53.010633 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.010590 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkv2k\" (UniqueName: \"kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k\") pod \"kas-connection-checker-685895c7c9-bzdtx\" (UID: \"41bb2e81-6a7b-434e-a6c4-184beef52d40\") " pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:45:53.010697 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.010664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jh9\" (UniqueName: \"kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:53.011187 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.011170 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:53.011471 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.011454 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5p78\" (UniqueName: \"kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:53.013606 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013581 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:45:53.013606 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013589 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:45:53.013754 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013626 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:45:53.013754 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013627 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:45:53.013846 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013803 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:45:53.013846 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013804 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:45:53.013846 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.013804 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:45:53.014103 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.014091 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:45:53.014432 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.014419 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:45:53.022403 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.022388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:45:53.042923 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.042779 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:45:53.049762 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.049535 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:45:53.049880 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.049776 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjgl9\"" Oct 30 18:45:53.049880 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.049811 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vvlg2\"" Oct 30 18:45:53.049880 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.049823 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:45:53.053034 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.053141 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.053141 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.053141 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.053317 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053236 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.053317 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.053317 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9hh\" (UniqueName: \"kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.053317 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053294 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.053495 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmlw\" (UniqueName: \"kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.053495 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.053579 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.053579 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.053579 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.053562 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.553541688 +0000 UTC m=+33.801814873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:45:53.053722 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.053722 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.053722 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.053722 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89cg\" (UniqueName: \"kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvt7\" (UniqueName: \"kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7\") pod \"volume-data-source-validator-58c9f5f46d-2tt7k\" (UID: \"7224a54e-ce8c-4b82-9247-11376266019b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.053900 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7k7\" (UniqueName: \"kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7\") pod \"network-check-source-6f78d59f99-f6bkw\" (UID: \"c4119de0-3ce3-4040-8ed3-a3c1366485da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:45:53.054189 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q5j\" (UniqueName: \"kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.054189 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.053974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766w5\" (UniqueName: \"kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.054189 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.054189 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.054385 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.054385 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054291 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.054385 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.054385 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4tm\" (UniqueName: \"kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.054385 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb455\" (UniqueName: \"kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054434 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.054586 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054563 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.054845 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.054601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.054845 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.054724 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:45:53.054845 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.054772 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.554756662 +0000 UTC m=+33.803029843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:45:53.055734 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.055118 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.055734 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.055691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.058382 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.058362 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.058493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.058382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.058493 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.058424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.058644 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.058625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.058679 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.058620 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.115682 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.113426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89cg\" (UniqueName: \"kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.115682 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.114331 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmlw\" (UniqueName: \"kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.116895 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.116423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q5j\" (UniqueName: \"kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.119233 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.117760 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4tm\" (UniqueName: \"kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.119233 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.118175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvt7\" (UniqueName: \"kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7\") pod \"volume-data-source-validator-58c9f5f46d-2tt7k\" (UID: \"7224a54e-ce8c-4b82-9247-11376266019b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:45:53.151370 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.151309 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:45:53.155055 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.155166 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155081 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155166 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155166 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155135 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7k7\" (UniqueName: \"kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7\") pod \"network-check-source-6f78d59f99-f6bkw\" (UID: \"c4119de0-3ce3-4040-8ed3-a3c1366485da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:45:53.155166 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766w5\" (UniqueName: \"kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb455\" (UniqueName: \"kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155431 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155481 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9hh\" (UniqueName: \"kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.155749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.156101 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.156101 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.156101 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.155893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.156307 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.156163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.156362 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.156319 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.156502 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.156504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.156541 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.156558 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.656540884 +0000 UTC m=+33.904814053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.156590 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.156602 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.156649 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:53.656637398 +0000 UTC m=+33.904910576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:45:53.156731 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.156730 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.157052 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.157031 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.157088 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.157058 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.158153 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.158130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.158335 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.158314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.158677 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.158660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.158838 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.158823 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.170724 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.170708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:45:53.188862 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.188845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:45:53.201525 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.201503 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:45:53.229305 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.229270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.230259 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.230180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.232199 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.232174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb455\" (UniqueName: \"kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.232716 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.232688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7k7\" (UniqueName: \"kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7\") pod \"network-check-source-6f78d59f99-f6bkw\" (UID: \"c4119de0-3ce3-4040-8ed3-a3c1366485da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:45:53.233910 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.233887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766w5\" (UniqueName: \"kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.235176 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.235152 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9hh\" (UniqueName: \"kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.245652 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.245293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:45:53.262191 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.262168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:45:53.262380 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.262366 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:53.292240 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.292127 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:45:53.313183 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.313085 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q9tbn\"" Oct 30 18:45:53.313318 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.313261 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:45:53.412151 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.412126 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-54zh4"] Oct 30 18:45:53.419351 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.419325 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497867c2_a03d_4717_b845_1d0ef71495d5.slice/crio-0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7 WatchSource:0}: Error finding container 0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7: Status 404 returned error can't find the container with id 0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7 Oct 30 18:45:53.427172 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.427150 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bzdtx"] Oct 30 18:45:53.432776 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.432751 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bb2e81_6a7b_434e_a6c4_184beef52d40.slice/crio-f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79 WatchSource:0}: Error finding container f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79: Status 404 returned error can't find the container with id f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79 Oct 30 18:45:53.440597 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.440562 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" event={"ID":"41bb2e81-6a7b-434e-a6c4-184beef52d40","Type":"ContainerStarted","Data":"f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79"} Oct 30 18:45:53.441800 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.441771 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-54zh4" event={"ID":"497867c2-a03d-4717-b845-1d0ef71495d5","Type":"ContainerStarted","Data":"0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7"} Oct 30 18:45:53.456103 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.454454 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fj8mt"] Oct 30 18:45:53.458937 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.458913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:53.459002 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.458956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:53.459061 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.458999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:53.459061 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.459018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:53.459193 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459138 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:45:53.459265 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459195 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.459177087 +0000 UTC m=+34.707450268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:45:53.459330 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459282 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:53.459330 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459320 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.459308921 +0000 UTC m=+34.707582099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:53.459483 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459385 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:53.459483 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459403 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:45:53.459483 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459435 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.459423781 +0000 UTC m=+34.707696960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:45:53.459627 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459494 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:45:53.459627 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.459525 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.459514629 +0000 UTC m=+34.707787795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:45:53.461502 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.461477 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c417afb_ca2e_4706_8004_f6651dd11f16.slice/crio-411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd WatchSource:0}: Error finding container 411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd: Status 404 returned error can't find the container with id 411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd Oct 30 18:45:53.533084 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.533042 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nlzxq"] Oct 30 18:45:53.545681 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.545656 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f91f0e_7db9_426d_ac87_4c055d52abaa.slice/crio-e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4 WatchSource:0}: Error finding container e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4: Status 404 returned error can't find the container with id e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4 Oct 30 18:45:53.553699 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.553678 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx"] Oct 30 18:45:53.559769 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.559745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.559841 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.559827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:53.559950 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.559937 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:45:53.559981 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.559961 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.55994477 +0000 UTC m=+34.808217935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:45:53.560013 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.559981 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.55997094 +0000 UTC m=+34.808244104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:45:53.560049 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.560008 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b20b3a_54d7_4dcd_b353_f9e28b7654c6.slice/crio-76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199 WatchSource:0}: Error finding container 76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199: Status 404 returned error can't find the container with id 76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199 Oct 30 18:45:53.572936 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.572913 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k"] Oct 30 18:45:53.601381 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.601204 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m"] Oct 30 18:45:53.607708 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.607688 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f732cf0_c45a_4146_9d07_63c3e1e15ba6.slice/crio-476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e WatchSource:0}: Error finding container 476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e: Status 404 returned error can't find the container with id 476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e Oct 30 18:45:53.649867 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.649842 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-95sk7"] Oct 30 18:45:53.653203 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.653186 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xw9bs"] Oct 30 18:45:53.657155 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.657136 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bd50e6_b8bb_4685_8bd7_2f1d0039fddc.slice/crio-a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7 WatchSource:0}: Error finding container a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7: Status 404 returned error can't find the container with id a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7 Oct 30 18:45:53.659177 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.659156 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8860e1_0ab3_4d6d_a343_1e8b4e7b3440.slice/crio-46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1 WatchSource:0}: Error finding container 46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1: Status 404 returned error can't find the container with id 46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1 Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.660287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.660405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.660419 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.660436 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.660520 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.660496758 +0000 UTC m=+34.908769936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:45:53.660580 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.660578 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:45:53.660827 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:53.660625 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:54.660609519 +0000 UTC m=+34.908882688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:45:53.684264 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:53.684243 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw"] Oct 30 18:45:53.690249 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:53.690231 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4119de0_3ce3_4040_8ed3_a3c1366485da.slice/crio-b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511 WatchSource:0}: Error finding container b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511: Status 404 returned error can't find the container with id b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511 Oct 30 18:45:54.064427 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.064401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:54.064584 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.064493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:54.064662 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.064645 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:54.064734 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.064713 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:26.06469501 +0000 UTC m=+66.312968177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:45:54.067551 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.067530 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:54.165399 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.165370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:54.167755 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.167731 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:54.191742 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.191718 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:45:54.262485 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.262461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:45:54.262616 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.262469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:54.313633 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.313605 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:45:54.313720 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.313643 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxtc\"" Oct 30 18:45:54.313720 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.313648 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgxcg\"" Oct 30 18:45:54.315701 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.315648 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:45:54.445233 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.445194 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xw9bs" event={"ID":"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440","Type":"ContainerStarted","Data":"46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1"} Oct 30 18:45:54.446506 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.446484 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" event={"ID":"8c417afb-ca2e-4706-8004-f6651dd11f16","Type":"ContainerStarted","Data":"53c2e04677fa9725377f656ae43e63496698ed117d09ea55da0aee817c1538bd"} Oct 30 18:45:54.446660 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.446517 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" event={"ID":"8c417afb-ca2e-4706-8004-f6651dd11f16","Type":"ContainerStarted","Data":"411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd"} Oct 30 18:45:54.447859 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.447838 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" event={"ID":"41bb2e81-6a7b-434e-a6c4-184beef52d40","Type":"ContainerStarted","Data":"14c5c2ad38a916ec50d86adc2fa6ac01b49d910fb00f614acd1167b604b4656a"} Oct 30 18:45:54.448893 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.448874 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" event={"ID":"1f732cf0-c45a-4146-9d07-63c3e1e15ba6","Type":"ContainerStarted","Data":"476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e"} Oct 30 18:45:54.449912 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.449894 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" event={"ID":"24b20b3a-54d7-4dcd-b353-f9e28b7654c6","Type":"ContainerStarted","Data":"76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199"} Oct 30 18:45:54.451227 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.451197 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-54zh4" event={"ID":"497867c2-a03d-4717-b845-1d0ef71495d5","Type":"ContainerStarted","Data":"2f294edf79c0c1b0b29ea0ebebae2eb3a804de21b1c6a1da7f3727fc0c252849"} Oct 30 18:45:54.452220 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.452195 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-95sk7" event={"ID":"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc","Type":"ContainerStarted","Data":"a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7"} Oct 30 18:45:54.453153 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.453136 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" event={"ID":"7224a54e-ce8c-4b82-9247-11376266019b","Type":"ContainerStarted","Data":"f72f4fdb7618a788f59ab9368dbc160378dd15ab13522fafdf2fe9234f9006f7"} Oct 30 18:45:54.454153 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.454126 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" event={"ID":"94f91f0e-7db9-426d-ac87-4c055d52abaa","Type":"ContainerStarted","Data":"e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4"} Oct 30 18:45:54.455075 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.455059 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" event={"ID":"c4119de0-3ce3-4040-8ed3-a3c1366485da","Type":"ContainerStarted","Data":"b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511"} Oct 30 18:45:54.467127 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.467107 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:54.467196 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.467142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:54.467196 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.467181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:54.467282 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.467240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:54.467282 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467268 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:54.467346 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467312 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:54.467346 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467316 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.467302508 +0000 UTC m=+36.715575671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:54.467346 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467322 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:45:54.467464 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467272 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:45:54.467464 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467356 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:45:54.467464 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467364 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.467350558 +0000 UTC m=+36.715623736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:45:54.467464 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467407 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.467381389 +0000 UTC m=+36.715654572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:45:54.467464 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.467424 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.467418009 +0000 UTC m=+36.715691173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:45:54.568642 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.568582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:54.568976 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.568676 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:45:54.568976 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.568766 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.568750846 +0000 UTC m=+36.817024023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:45:54.568976 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.568829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:54.569122 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.569024 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.569006337 +0000 UTC m=+36.817279514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:45:54.605306 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.605007 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnc44"] Oct 30 18:45:54.612457 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:54.612156 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd95a5_32ae_4c7a_a067_bc40ad4bc826.slice/crio-dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e WatchSource:0}: Error finding container dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e: Status 404 returned error can't find the container with id dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e Oct 30 18:45:54.669573 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.669551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:54.669689 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.669658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:54.669751 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.669702 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:45:54.669802 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.669758 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.669741413 +0000 UTC m=+36.918014594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:45:54.669865 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.669808 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:54.669865 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.669823 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:45:54.669944 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:54.669868 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:45:56.66985287 +0000 UTC m=+36.918126049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:45:54.694236 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.694198 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lbk7g"] Oct 30 18:45:54.700001 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:45:54.699964 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ee20f_a7f2_4994_abea_4c233d8b8da3.slice/crio-764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680 WatchSource:0}: Error finding container 764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680: Status 404 returned error can't find the container with id 764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680 Oct 30 18:45:54.728002 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.727937 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" podStartSLOduration=366.727922299 podStartE2EDuration="6m6.727922299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:54.727583949 +0000 UTC m=+34.975857136" watchObservedRunningTime="2026-10-30 18:45:54.727922299 +0000 UTC m=+34.976195486" Oct 30 18:45:54.952055 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.952010 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-54zh4" podStartSLOduration=366.95199588 podStartE2EDuration="6m6.95199588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:54.840133359 +0000 UTC m=+35.088406547" watchObservedRunningTime="2026-10-30 18:45:54.95199588 +0000 UTC m=+35.200269060" Oct 30 18:45:54.952806 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:54.952764 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" podStartSLOduration=366.952753948 podStartE2EDuration="6m6.952753948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:54.951585423 +0000 UTC m=+35.199858610" watchObservedRunningTime="2026-10-30 18:45:54.952753948 +0000 UTC m=+35.201027136" Oct 30 18:45:55.465425 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:55.465317 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnc44" event={"ID":"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826","Type":"ContainerStarted","Data":"dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e"} Oct 30 18:45:55.470397 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:55.469998 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lbk7g" event={"ID":"bb7ee20f-a7f2-4994-abea-4c233d8b8da3","Type":"ContainerStarted","Data":"764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680"} Oct 30 18:45:55.537650 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:55.537530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.757544 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:55.757344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:56.000945 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.000853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.488259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.488321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.488398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.488433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.488662 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.488725 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.488707228 +0000 UTC m=+40.736980415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.489111 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.489170 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.489152351 +0000 UTC m=+40.737425522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.489292 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:45:56.489356 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.489331 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.489318584 +0000 UTC m=+40.737591763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:45:56.490389 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.490252 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:56.490389 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.490273 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:45:56.490389 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.490319 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.490304906 +0000 UTC m=+40.738578076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:45:56.589770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.589352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:56.589770 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.589439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:45:56.589770 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.589632 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.589615302 +0000 UTC m=+40.837888471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:45:56.589770 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.589708 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:45:56.589770 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.589741 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.589729431 +0000 UTC m=+40.838002598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:45:56.687579 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.687547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:56.690351 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.690325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:45:56.690460 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.690424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:45:56.690670 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.690655 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:45:56.690746 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.690681 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:45:56.690746 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.690726 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.690715032 +0000 UTC m=+40.938988196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:45:56.690859 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.690790 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:45:56.690859 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:45:56.690824 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:00.690812856 +0000 UTC m=+40.939086021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:45:56.883985 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:56.883753 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:57.493292 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:57.492388 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" event={"ID":"7224a54e-ce8c-4b82-9247-11376266019b","Type":"ContainerStarted","Data":"943c856fabe030dc863305093758a50a5cf43305b6f89f71d7267cb8653f68b6"} Oct 30 18:45:57.621033 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:57.620978 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" podStartSLOduration=339.25204215 podStartE2EDuration="5m42.620963459s" totalImagesPullingTime="3.368921309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:57.620241591 +0000 UTC m=+37.868514778" watchObservedRunningTime="2026-10-30 18:45:57.620963459 +0000 UTC m=+37.869236664" Oct 30 18:45:57.780998 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:57.780730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:57.961449 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:57.961388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:59.504480 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:59.503123 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/0.log" Oct 30 18:45:59.504480 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:59.503364 2573 generic.go:356] "Generic (PLEG): container finished" podID="94f91f0e-7db9-426d-ac87-4c055d52abaa" containerID="44f63e23427bfa2553908d72660e425a7f0ec6861ee2dea502e97acc6b336984" exitCode=255 Oct 30 18:45:59.504480 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:59.503437 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" event={"ID":"94f91f0e-7db9-426d-ac87-4c055d52abaa","Type":"ContainerDied","Data":"44f63e23427bfa2553908d72660e425a7f0ec6861ee2dea502e97acc6b336984"} Oct 30 18:45:59.504480 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:45:59.503772 2573 scope.go:122] "RemoveContainer" containerID="44f63e23427bfa2553908d72660e425a7f0ec6861ee2dea502e97acc6b336984" Oct 30 18:46:00.513285 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.510061 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" event={"ID":"1f732cf0-c45a-4146-9d07-63c3e1e15ba6","Type":"ContainerStarted","Data":"71610c0ce5efbe42abfec9a7de398d32c6d635a60da28ca501b9d5d9425bd0f2"} Oct 30 18:46:00.513285 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.513022 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" event={"ID":"24b20b3a-54d7-4dcd-b353-f9e28b7654c6","Type":"ContainerStarted","Data":"f2dff396677e4795d07e708fed3e1c3461ce9d35b65ad9645ad67b983cd1a5ff"} Oct 30 18:46:00.515627 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.515603 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-95sk7" event={"ID":"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc","Type":"ContainerStarted","Data":"57dde4d66137ce3ce7aabce092778bf123daadc97e446708042aa7a453f237ae"} Oct 30 18:46:00.518560 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.518536 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:00.519004 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.518984 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/0.log" Oct 30 18:46:00.519101 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.519025 2573 generic.go:356] "Generic (PLEG): container finished" podID="94f91f0e-7db9-426d-ac87-4c055d52abaa" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" exitCode=255 Oct 30 18:46:00.519297 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.519274 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" event={"ID":"94f91f0e-7db9-426d-ac87-4c055d52abaa","Type":"ContainerDied","Data":"6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a"} Oct 30 18:46:00.519378 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.519339 2573 scope.go:122] "RemoveContainer" containerID="44f63e23427bfa2553908d72660e425a7f0ec6861ee2dea502e97acc6b336984" Oct 30 18:46:00.519378 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.519356 2573 scope.go:122] "RemoveContainer" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" Oct 30 18:46:00.519519 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.519501 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:00.523182 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.523158 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xw9bs" event={"ID":"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440","Type":"ContainerStarted","Data":"d64807b6330180394a50414a801f39090a972acb2b69c3c4cc42f1f7dcf314e2"} Oct 30 18:46:00.531228 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.531193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:00.531322 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.531257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:00.531322 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.531313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:00.531436 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.531344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:00.531436 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531387 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:00.531540 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531451 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.531433471 +0000 UTC m=+48.779706645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:00.531540 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531506 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:00.531540 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531524 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:00.531540 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531534 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:00.531540 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531542 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.531530808 +0000 UTC m=+48.779803975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:00.531799 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531567 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.531556422 +0000 UTC m=+48.779829589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:00.531799 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531598 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:00.531799 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.531639 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.531628584 +0000 UTC m=+48.779901764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:00.625774 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.625710 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" podStartSLOduration=339.782871692 podStartE2EDuration="5m45.625690593s" totalImagesPullingTime="5.842818901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:00.624572743 +0000 UTC m=+40.872845932" watchObservedRunningTime="2026-10-30 18:46:00.625690593 +0000 UTC m=+40.873963779" Oct 30 18:46:00.632913 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.632006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:00.632913 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.632187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:00.639626 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.634952 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:46:00.639771 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.636135 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.636113311 +0000 UTC m=+48.884386482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:00.639771 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.639695 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.639678043 +0000 UTC m=+48.887951211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:46:00.733433 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.733401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:00.733599 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.733528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:00.733599 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.733549 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:46:00.733720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.733605 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.733588428 +0000 UTC m=+48.981861617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:46:00.733720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.733648 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:00.733720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.733657 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:00.733720 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:00.733683 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.733675614 +0000 UTC m=+48.981948781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:00.854220 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:00.853265 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" podStartSLOduration=339.516014758 podStartE2EDuration="5m45.853251583s" totalImagesPullingTime="6.337236825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:00.852752906 +0000 UTC m=+41.101026092" watchObservedRunningTime="2026-10-30 18:46:00.853251583 +0000 UTC m=+41.101524768" Oct 30 18:46:01.023036 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.022967 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-95sk7" podStartSLOduration=340.243797614 podStartE2EDuration="5m46.022949743s" totalImagesPullingTime="5.779152129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:00.971979792 +0000 UTC m=+41.220252977" watchObservedRunningTime="2026-10-30 18:46:01.022949743 +0000 UTC m=+41.271222930" Oct 30 18:46:01.024304 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.024278 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2ckq9"] Oct 30 18:46:01.044661 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.044631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.085916 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.085893 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ckq9"] Oct 30 18:46:01.088492 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.088443 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xw9bs" podStartSLOduration=2.502622428 podStartE2EDuration="9.088428452s" totalImagesPullingTime="6.585806024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:01.086911846 +0000 UTC m=+41.335185032" watchObservedRunningTime="2026-10-30 18:46:01.088428452 +0000 UTC m=+41.336701639" Oct 30 18:46:01.101420 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.101375 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:46:01.101540 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.101477 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k86kp\"" Oct 30 18:46:01.110563 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.110509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.134688 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.134651 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:46:01.239957 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.239917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md79n\" (UniqueName: \"kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.240136 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.240026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.240136 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.240123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.240269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.240251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.240326 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.240273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.275640 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.275600 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:46:01.290276 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.290251 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.297581 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.297487 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.318057 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.318034 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv"] Oct 30 18:46:01.337807 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.337786 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w"] Oct 30 18:46:01.337949 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.337931 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:01.340956 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.340935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341052 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.340970 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341142 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.341125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md79n\" (UniqueName: \"kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341257 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.341140 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:46:01.341257 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.341152 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341257 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.341202 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:01.841182733 +0000 UTC m=+42.089455903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : configmap "kube-rbac-proxy" not found Oct 30 18:46:01.341477 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.341454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341558 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.341488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.341636 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.341621 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:01.341710 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.341675 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:01.841659731 +0000 UTC m=+42.089932899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:01.341901 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.341877 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.356269 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.356203 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:46:01.356365 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.356323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:01.393942 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.393908 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:46:01.394056 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.393946 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:01.396053 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.396032 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w"] Oct 30 18:46:01.404043 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.404022 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv"] Oct 30 18:46:01.407110 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.406880 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qqsp5\"" Oct 30 18:46:01.417515 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.416947 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md79n\" (UniqueName: \"kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.442749 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.442723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.442954 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.442924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.443106 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.443085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.443190 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.443138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftb8j\" (UniqueName: \"kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j\") pod \"migrator-66df5d496b-g22qv\" (UID: \"bdccbf5a-017a-46f7-a82e-df13cc459a30\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:01.443190 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.443182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.469853 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.469823 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.529724 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.529698 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:01.530654 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.530204 2573 scope.go:122] "RemoveContainer" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" Oct 30 18:46:01.530654 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.530554 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:01.533673 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.533590 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" event={"ID":"c4119de0-3ce3-4040-8ed3-a3c1366485da","Type":"ContainerStarted","Data":"982ac0a07a02e73ba0698522e1d09008ed736d70395aabd1182c7498ec848240"} Oct 30 18:46:01.543641 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.543741 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftb8j\" (UniqueName: \"kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j\") pod \"migrator-66df5d496b-g22qv\" (UID: \"bdccbf5a-017a-46f7-a82e-df13cc459a30\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:01.543881 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.543999 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfz9\" (UniqueName: \"kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9\") pod \"migrator-66df5d496b-nvg6w\" (UID: \"48d1db69-50e4-4cc9-a744-2ab11bd3f519\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:01.543999 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.543999 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.543977 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.544132 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.544056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.544638 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.544612 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.544903 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.544881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.604168 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.604146 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.604296 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.604179 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftb8j\" (UniqueName: \"kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j\") pod \"migrator-66df5d496b-g22qv\" (UID: \"bdccbf5a-017a-46f7-a82e-df13cc459a30\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:01.622624 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.622563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:01.644887 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.644855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfz9\" (UniqueName: \"kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9\") pod \"migrator-66df5d496b-nvg6w\" (UID: \"48d1db69-50e4-4cc9-a744-2ab11bd3f519\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:01.663818 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.663791 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:01.666241 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.666166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.704600 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.704574 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfz9\" (UniqueName: \"kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9\") pod \"migrator-66df5d496b-nvg6w\" (UID: \"48d1db69-50e4-4cc9-a744-2ab11bd3f519\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:01.843583 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.843310 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.847450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.847561 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.847568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.847623 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:02.847603138 +0000 UTC m=+43.095876319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.847662 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:46:01.847783 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:01.847716 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:02.847699021 +0000 UTC m=+43.095972201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : configmap "kube-rbac-proxy" not found Oct 30 18:46:01.984160 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:01.984129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:02.093093 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.093060 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:46:02.099959 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:46:02.099934 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361aa627_cb6e_4b51_96b4_f66cc3ebb23a.slice/crio-6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a WatchSource:0}: Error finding container 6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a: Status 404 returned error can't find the container with id 6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a Oct 30 18:46:02.144956 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.144938 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv"] Oct 30 18:46:02.154641 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:46:02.154618 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdccbf5a_017a_46f7_a82e_df13cc459a30.slice/crio-837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b WatchSource:0}: Error finding container 837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b: Status 404 returned error can't find the container with id 837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b Oct 30 18:46:02.374873 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.374810 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w"] Oct 30 18:46:02.381758 ip-10-0-136-46 kubenswrapper[2573]: W1030 18:46:02.381730 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d1db69_50e4_4cc9_a744_2ab11bd3f519.slice/crio-c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8 WatchSource:0}: Error finding container c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8: Status 404 returned error can't find the container with id c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8 Oct 30 18:46:02.539297 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.539261 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" event={"ID":"361aa627-cb6e-4b51-96b4-f66cc3ebb23a","Type":"ContainerStarted","Data":"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c"} Oct 30 18:46:02.539705 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.539317 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" event={"ID":"361aa627-cb6e-4b51-96b4-f66cc3ebb23a","Type":"ContainerStarted","Data":"6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a"} Oct 30 18:46:02.541043 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.541017 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" event={"ID":"bdccbf5a-017a-46f7-a82e-df13cc459a30","Type":"ContainerStarted","Data":"837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b"} Oct 30 18:46:02.542723 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.542697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lbk7g" event={"ID":"bb7ee20f-a7f2-4994-abea-4c233d8b8da3","Type":"ContainerStarted","Data":"365a3e2b7872954d1dd48b977da81ba345a0e0f3cde85de505c02cacba92ada8"} Oct 30 18:46:02.542841 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.542803 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:46:02.544073 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.544048 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" event={"ID":"48d1db69-50e4-4cc9-a744-2ab11bd3f519","Type":"ContainerStarted","Data":"c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8"} Oct 30 18:46:02.655559 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.655444 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" podStartSLOduration=1.655424217 podStartE2EDuration="1.655424217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:02.65424569 +0000 UTC m=+42.902518877" watchObservedRunningTime="2026-10-30 18:46:02.655424217 +0000 UTC m=+42.903697406" Oct 30 18:46:02.661522 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.660188 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8npkv"] Oct 30 18:46:02.682372 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.682348 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.742776 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.742754 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:46:02.743341 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.743326 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4rzs\"" Oct 30 18:46:02.745705 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.745689 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:46:02.750815 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.750794 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8npkv"] Oct 30 18:46:02.760602 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.760583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86rq\" (UniqueName: \"kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.760679 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.760646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.760730 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.760713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.773384 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.773350 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lbk7g" podStartSLOduration=35.748536363 podStartE2EDuration="42.773339153s" totalImagesPullingTime="7.02480279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:02.772706309 +0000 UTC m=+43.020979495" watchObservedRunningTime="2026-10-30 18:46:02.773339153 +0000 UTC m=+43.021612339" Oct 30 18:46:02.838849 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.838829 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:46:02.838849 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.838829 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:46:02.861321 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:02.861407 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.861407 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.861473 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:02.861518 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86rq\" (UniqueName: \"kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.861571 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:02.861555 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 18:46:02.861622 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:02.861612 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:04.861596947 +0000 UTC m=+45.109870110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : configmap "kube-rbac-proxy" not found Oct 30 18:46:02.861673 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:02.861643 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:02.861708 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:02.861692 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:04.8616773 +0000 UTC m=+45.109950471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:02.861851 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.861836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.865229 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.865181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:02.887070 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.887028 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" podStartSLOduration=338.154325181 podStartE2EDuration="5m45.887018241s" totalImagesPullingTime="7.73269306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:02.886332422 +0000 UTC m=+43.134605608" watchObservedRunningTime="2026-10-30 18:46:02.887018241 +0000 UTC m=+43.135291427" Oct 30 18:46:02.906485 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.906439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:02.920242 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:02.920202 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86rq\" (UniqueName: \"kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:03.010804 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:03.010776 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:03.079110 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:03.078952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:03.152178 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:03.152150 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:03.152301 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:03.152281 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:03.152595 ip-10-0-136-46 kubenswrapper[2573]: I1030 18:46:03.152581 2573 scope.go:122] "RemoveContainer" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" Oct 30 18:46:03.152782 ip-10-0-136-46 kubenswrapper[2573]: E1030 18:46:03.152764 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:03.443605 ip-10-0-136-46 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:46:03.450239 ip-10-0-136-46 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:46:03.450468 ip-10-0-136-46 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:46:03.450677 ip-10-0-136-46 systemd[1]: kubelet.service: Consumed 3.026s CPU time. Oct 30 18:46:03.461990 ip-10-0-136-46 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:46:03.569703 ip-10-0-136-46 kubenswrapper[4789]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:46:03.569703 ip-10-0-136-46 kubenswrapper[4789]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:46:03.569703 ip-10-0-136-46 kubenswrapper[4789]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:46:03.569703 ip-10-0-136-46 kubenswrapper[4789]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:46:03.571796 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571704 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.571796 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571794 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.571796 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571798 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.571796 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571802 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571805 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571807 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571810 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571813 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571815 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571818 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571821 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571823 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571826 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571829 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571837 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571840 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571843 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571845 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571848 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571851 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571853 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571856 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571859 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.571945 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571861 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571865 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571870 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571873 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571875 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571878 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571881 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571884 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571886 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571891 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571894 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571898 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571902 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571906 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571911 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571916 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571920 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571924 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571928 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.572570 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571931 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571935 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571940 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571943 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571946 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571950 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571953 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571955 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571958 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571961 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571964 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571966 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571969 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571972 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571974 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571977 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571979 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571982 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571985 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.573065 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571988 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571994 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.571999 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572003 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572008 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572010 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572014 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572016 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572019 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572022 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572025 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572027 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572031 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572034 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572036 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572039 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572041 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572044 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572047 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572050 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.573549 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572052 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572055 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572057 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572060 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572062 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572065 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572067 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572070 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572075 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572078 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572081 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572083 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572086 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572088 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572092 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572094 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572097 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572100 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572104 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.574040 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572108 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572114 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572118 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572122 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572126 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.572129 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573002 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573011 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573015 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573018 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573021 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573023 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573026 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573029 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573032 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573035 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573037 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573040 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573043 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573045 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.574528 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573048 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573053 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573056 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573059 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573061 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573065 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573070 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573073 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573077 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573079 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573082 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573085 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573087 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573090 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573093 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573095 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573098 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573100 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573103 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.575036 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573106 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573109 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573111 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573114 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573117 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573119 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573122 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573125 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573127 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573130 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573132 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573135 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573137 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573140 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573144 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573147 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573149 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573152 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573154 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573157 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.575562 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573161 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573163 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573166 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573168 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573171 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573173 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573176 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573178 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573181 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573184 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573186 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573189 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573191 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573194 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573196 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573199 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573202 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573204 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573223 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573226 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.576062 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573229 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573231 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573235 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573238 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573240 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573243 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573247 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573250 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573252 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573256 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573260 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573264 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573267 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573269 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573272 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573274 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573277 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573280 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573283 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.576735 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573285 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573288 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573290 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573293 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573296 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573298 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573301 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573303 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573306 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573308 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573311 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573313 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573317 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573320 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573885 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573891 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573894 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573898 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573901 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.577267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573906 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573909 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573912 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573914 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573917 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573920 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573922 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573925 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573928 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573930 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573933 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573935 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573938 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573940 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573943 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573946 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573948 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573951 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573953 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573956 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.577723 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573958 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573961 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573963 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573966 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573969 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573971 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573974 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573977 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573980 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573983 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573985 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573988 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573992 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573995 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.573998 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574000 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574003 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574005 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574008 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.578235 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574010 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574013 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574015 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574018 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574020 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574024 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574027 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574029 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574032 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574034 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574037 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574041 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574045 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574048 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574051 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574054 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574057 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574059 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574062 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.578876 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574064 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574067 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574070 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574073 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574075 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574078 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574081 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574083 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574086 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574088 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574091 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574093 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574096 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574098 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574101 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574104 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574106 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574108 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574111 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574113 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.579382 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574116 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574118 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574121 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574123 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574126 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574128 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574131 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574134 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574136 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574139 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574141 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574144 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574146 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574148 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574151 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574157 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574162 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574165 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574168 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.579865 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574171 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574174 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574177 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574180 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574279 4789 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574287 4789 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574297 4789 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574302 4789 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574307 4789 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574310 4789 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574314 4789 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574319 4789 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574322 4789 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574325 4789 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574329 4789 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574332 4789 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574335 4789 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574338 4789 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574341 4789 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574345 4789 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574348 4789 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574351 4789 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574356 4789 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:46:03.580337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574359 4789 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574362 4789 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574365 4789 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574368 4789 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574373 4789 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574376 4789 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574379 4789 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574383 4789 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574386 4789 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574389 4789 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574392 4789 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574395 4789 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574398 4789 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574402 4789 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574405 4789 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574408 4789 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574411 4789 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574414 4789 flags.go:64] FLAG: --enable-server="true" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574417 4789 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574439 4789 flags.go:64] FLAG: --event-burst="100" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574443 4789 flags.go:64] FLAG: --event-qps="50" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574447 4789 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574450 4789 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574453 4789 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:46:03.580909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574461 4789 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574464 4789 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574467 4789 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574470 4789 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574473 4789 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574476 4789 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574479 4789 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574482 4789 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574485 4789 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574488 4789 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574491 4789 flags.go:64] FLAG: --feature-gates="" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574495 4789 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574498 4789 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574501 4789 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574504 4789 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574508 4789 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574511 4789 flags.go:64] FLAG: --help="false" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574514 4789 flags.go:64] FLAG: --hostname-override="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574517 4789 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574520 4789 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574523 4789 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574526 4789 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574530 4789 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:46:03.581507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574533 4789 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574536 4789 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574539 4789 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574542 4789 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574545 4789 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574548 4789 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574551 4789 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574554 4789 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574557 4789 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574560 4789 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574565 4789 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574568 4789 flags.go:64] FLAG: --lock-file="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574571 4789 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574575 4789 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574579 4789 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574584 4789 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574587 4789 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574590 4789 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574593 4789 flags.go:64] FLAG: --logging-format="text" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574596 4789 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574599 4789 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574602 4789 flags.go:64] FLAG: --manifest-url="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574606 4789 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574610 4789 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574614 4789 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:46:03.582252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574619 4789 flags.go:64] FLAG: --max-pods="110" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574622 4789 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574625 4789 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574628 4789 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574631 4789 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574634 4789 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574637 4789 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574641 4789 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574647 4789 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574650 4789 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574653 4789 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574656 4789 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574659 4789 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574662 4789 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574665 4789 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574668 4789 flags.go:64] FLAG: --port="10250" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574671 4789 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574675 4789 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0b612e7ced5006ab4" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574679 4789 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574682 4789 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574686 4789 flags.go:64] FLAG: --register-node="true" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574689 4789 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574693 4789 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574696 4789 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574699 4789 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574702 4789 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:46:03.583110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574706 4789 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574709 4789 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574712 4789 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574715 4789 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574718 4789 flags.go:64] FLAG: --runonce="false" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574721 4789 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574724 4789 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574727 4789 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574731 4789 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574733 4789 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574737 4789 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574740 4789 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574743 4789 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574746 4789 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574749 4789 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574752 4789 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574755 4789 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574758 4789 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574761 4789 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574763 4789 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574769 4789 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574771 4789 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574774 4789 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574781 4789 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574784 4789 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:46:03.583786 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574787 4789 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574791 4789 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574794 4789 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574798 4789 flags.go:64] FLAG: --v="2" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574802 4789 flags.go:64] FLAG: --version="false" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574806 4789 flags.go:64] FLAG: --vmodule="" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574810 4789 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.574813 4789 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574954 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574959 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574962 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574965 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574968 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574971 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574973 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574976 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574979 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574982 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574985 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574987 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574990 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.584560 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574993 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574995 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.574998 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575000 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575003 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575006 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575009 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575013 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575016 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575021 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575023 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575026 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575032 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575035 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575038 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575040 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575043 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575046 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575049 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.585069 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575052 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575054 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575057 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575059 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575062 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575065 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575068 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575070 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575074 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575078 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575081 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575085 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575088 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575091 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575093 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575096 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575098 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575101 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575103 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.585646 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575106 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575108 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575111 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575115 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575117 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575120 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575124 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575126 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575129 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575131 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575134 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575137 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575139 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575142 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575145 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575147 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575150 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575152 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575155 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575157 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575160 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.586158 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575163 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575166 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575168 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575171 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575173 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575176 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575179 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575181 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575184 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575186 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575189 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575191 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575194 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575196 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575200 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575203 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575218 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575222 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575225 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.586755 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575228 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575231 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575234 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575236 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575239 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575241 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575244 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575246 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575250 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575252 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575255 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575257 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575260 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575262 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575265 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.575292 4789 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:46:03.587267 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575327 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575331 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575335 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575338 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575340 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575343 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575346 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575348 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575351 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575354 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575356 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575361 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575363 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575366 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575368 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575373 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575376 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575379 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575381 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575384 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.587696 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575386 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575389 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575399 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575402 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575404 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575407 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575409 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575412 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575415 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575418 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575420 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575423 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575426 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575429 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575432 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575435 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575437 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575440 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575443 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.588182 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575445 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575448 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575450 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575453 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575457 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575459 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575462 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575464 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575468 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575471 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575474 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575476 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575479 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575481 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575484 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575487 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575489 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575492 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575494 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575497 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.588737 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575500 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575502 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575505 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575507 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575510 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575513 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575516 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575519 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575523 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575527 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575530 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575532 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575535 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575538 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575540 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575543 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575547 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575550 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575552 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.589319 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575555 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575559 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575561 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575564 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575567 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575570 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575572 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575575 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575578 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575582 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575585 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575588 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575592 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575595 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575597 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575600 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575602 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575605 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575608 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.589832 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575610 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575613 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575616 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575618 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575621 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575624 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575626 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575638 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.575640 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.580337 4789 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.580352 4789 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.580371 4789 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.580376 4789 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580451 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580456 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580460 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580464 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580467 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580470 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580473 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.590337 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580475 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580478 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580481 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580483 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580486 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580488 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580491 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580494 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580496 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580499 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580502 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580505 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580507 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580510 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580514 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580517 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580519 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580522 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580525 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.590830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580529 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580532 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580535 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580538 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580541 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580543 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580546 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580549 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580552 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580554 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580557 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580560 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580563 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580565 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580568 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580570 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580573 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580575 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580578 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580580 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.591384 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580583 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580586 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580588 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580591 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580594 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580597 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580600 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580602 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580605 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580608 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580611 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580613 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580616 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580618 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580621 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580623 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580627 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580631 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580633 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.591947 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580636 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580639 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580642 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580645 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580647 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580650 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580652 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580655 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580657 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580660 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580663 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580665 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580668 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580671 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580673 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580676 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580679 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580681 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580684 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580686 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.592488 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580689 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580691 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580694 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580697 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580699 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580702 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580704 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580707 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580709 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580712 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580714 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580717 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580720 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580722 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580725 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580727 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580730 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580733 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580735 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580738 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.592962 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580740 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.580766 4789 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580905 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580909 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580912 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580915 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580917 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580920 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580923 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580925 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580928 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580931 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580933 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580936 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580938 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580941 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.593450 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580944 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580946 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580949 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580952 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580954 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580957 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580959 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580961 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580964 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580967 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580969 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580971 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580974 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580977 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580979 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580982 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580984 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580987 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580989 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.593829 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580992 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580994 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580996 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.580999 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581002 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581004 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581006 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581009 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581013 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581017 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581020 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581023 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581025 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581028 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581031 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581034 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581036 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581039 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581041 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.594377 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581044 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581046 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581049 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581052 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581054 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581057 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581059 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581062 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581064 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581067 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581069 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581072 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581074 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581077 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581079 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581082 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581084 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581087 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581089 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581092 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.594830 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581094 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581097 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581100 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581102 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581105 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581107 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581110 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581113 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581116 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581118 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581121 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581124 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581126 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581129 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581132 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581136 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581138 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581141 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581144 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.595325 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581146 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581149 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581151 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581153 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581156 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581158 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581161 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581164 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581166 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581169 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581171 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581174 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581176 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581178 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581181 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.581204 4789 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:46:03.595822 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581262 4789 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581268 4789 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581271 4789 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581274 4789 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581277 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581281 4789 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581284 4789 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581286 4789 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581289 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581292 4789 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581294 4789 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581296 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581299 4789 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581302 4789 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581304 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581307 4789 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581309 4789 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581312 4789 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581314 4789 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:46:03.596221 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581317 4789 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581319 4789 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581322 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581324 4789 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581327 4789 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581329 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581332 4789 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581337 4789 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581339 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581342 4789 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581344 4789 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581347 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581350 4789 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581352 4789 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581356 4789 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581360 4789 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581363 4789 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581366 4789 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581368 4789 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:46:03.596680 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581371 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581374 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581376 4789 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581379 4789 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581381 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581384 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581387 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581389 4789 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581392 4789 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581394 4789 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581397 4789 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581399 4789 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581402 4789 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581405 4789 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581407 4789 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581410 4789 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581413 4789 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581415 4789 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581418 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581421 4789 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:46:03.597177 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581425 4789 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581428 4789 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581431 4789 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581433 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581436 4789 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581439 4789 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581441 4789 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581444 4789 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581446 4789 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581449 4789 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581452 4789 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581454 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581456 4789 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581460 4789 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581462 4789 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581465 4789 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581467 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581470 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581472 4789 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581474 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:46:03.597697 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581477 4789 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581479 4789 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581482 4789 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581485 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581487 4789 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581489 4789 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581492 4789 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581494 4789 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581497 4789 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581499 4789 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581502 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581504 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581508 4789 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581511 4789 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581515 4789 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581518 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581522 4789 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581526 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581529 4789 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:46:03.598411 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581532 4789 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581535 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581538 4789 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581541 4789 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581544 4789 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581547 4789 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581550 4789 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581553 4789 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:03.581555 4789 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.581604 4789 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.583620 4789 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.583671 4789 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.583853 4789 server.go:1014] "Starting client certificate rotation" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.584000 4789 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.584068 4789 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:40:20 +0000 UTC" deadline="2028-05-30 17:16:00.799680321 +0000 UTC" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.584116 4789 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13870h29m57.215567667s" Oct 30 18:46:03.598886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.584306 4789 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:46:03.599285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.585366 4789 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:46:03.599285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.586801 4789 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:46:03.599285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.588557 4789 log.go:25] "Validated CRI v1 image API" Oct 30 18:46:03.599285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.589257 4789 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:46:03.599285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.591158 4789 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 88d613c0-f86d-4437-9193-370eed9ec610:/dev/nvme0n1p3 fbc9e2df-684a-40ad-a8fa-bbaa0746e0cd:/dev/nvme0n1p4] Oct 30 18:46:03.599460 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.591179 4789 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea62417ad8a443cfebe994240cc15c7c143327c8ae15b463c47fe0d6e62bc95d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea62417ad8a443cfebe994240cc15c7c143327c8ae15b463c47fe0d6e62bc95d/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f72f4fdb7618a788f59ab9368dbc160378dd15ab13522fafdf2fe9234f9006f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f72f4fdb7618a788f59ab9368dbc160378dd15ab13522fafdf2fe9234f9006f7/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~projected/kube-api-access-gc4tm:{mountpoint:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~projected/kube-api-access-gc4tm major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d3eacfd-ea78-48c0-8968-da01c34ec561/volumes/kubernetes.io~projected/kube-api-access-hrnhg:{mountpoint:/var/lib/kubelet/pods/0d3eacfd-ea78-48c0-8968-da01c34ec561/volumes/kubernetes.io~projected/kube-api-access-hrnhg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/150b998b-9608-4846-ac54-8a8ec8a7c620/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/150b998b-9608-4846-ac54-8a8ec8a7c620/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~projected/kube-api-access-l4q5j:{mountpoint:/var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~projected/kube-api-access-l4q5j major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~projected/kube-api-access-v89cg:{mountpoint:/var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~projected/kube-api-access-v89cg major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f076db4-f03f-4bf3-be39-e899370f953c/volumes/kubernetes.io~projected/kube-api-access-pzcm7:{mountpoint:/var/lib/kubelet/pods/2f076db4-f03f-4bf3-be39-e899370f953c/volumes/kubernetes.io~projected/kube-api-access-pzcm7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/361aa627-cb6e-4b51-96b4-f66cc3ebb23a/volumes/kubernetes.io~projected/kube-api-access-zq882:{mountpoint:/var/lib/kubelet/pods/361aa627-cb6e-4b51-96b4-f66cc3ebb23a/volumes/kubernetes.io~projected/kube-api-access-zq882 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38e48c7c-d60b-4daf-aa1a-b4578e3d910f/volumes/kubernetes.io~projected/kube-api-access-stdln:{mountpoint:/var/lib/kubelet/pods/38e48c7c-d60b-4daf-aa1a-b4578e3d910f/volumes/kubernetes.io~projected/kube-api-access-stdln major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41bb2e81-6a7b-434e-a6c4-184beef52d40/volumes/kubernetes.io~projected/kube-api-access-pkv2k:{mountpoint:/var/lib/kubelet/pods/41bb2e81-6a7b-434e-a6c4-184beef52d40/volumes/kubernetes.io~projected/kube-api-access-pkv2k major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48d1db69-50e4-4cc9-a744-2ab11bd3f519/volumes/kubernetes.io~projected/kube-api-access-2lfz9:{mountpoint:/var/lib/kubelet/pods/48d1db69-50e4-4cc9-a744-2ab11bd3f519/volumes/kubernetes.io~projected/kube-api-access-2lfz9 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/497867c2-a03d-4717-b845-1d0ef71495d5/volumes/kubernetes.io~projected/kube-api-access-t99zj:{mountpoint:/var/lib/kubelet/pods/497867c2-a03d-4717-b845-1d0ef71495d5/volumes/kubernetes.io~projected/kube-api-access-t99zj major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~projected/kube-api-access-sljzq:{mountpoint:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~projected/kube-api-access-sljzq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/kube-api-access-dthhg:{mountpoint:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/kube-api-access-dthhg major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7224a54e-ce8c-4b82-9247-11376266019b/volumes/kubernetes.io~projected/kube-api-access-nnvt7:{mountpoint:/var/lib/kubelet/pods/7224a54e-ce8c-4b82-9247-11376266019b/volumes/kubernetes.io~projected/kube-api-access-nnvt7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cee29d4-8788-4842-b8c5-a2f6e6f3996a/volumes/kubernetes.io~projected/kube-api-access-2g6px:{mountpoint:/var/lib/kubelet/pods/7cee29d4-8788-4842-b8c5-a2f6e6f3996a/volumes/kubernetes.io~projected/kube-api-access-2g6px major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fd86c83-2fdf-408f-a2f3-507b654c1f45/volumes/kubernetes.io~projected/kube-api-access-md79n:{mountpoint:/var/lib/kubelet/pods/7fd86c83-2fdf-408f-a2f3-507b654c1f45/volumes/kubernetes.io~projected/kube-api-access-md79n major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b36189e-0ae0-498e-8bd9-0391d1f3a47c/volumes/kubernetes.io~projected/kube-api-access-cw889:{mountpoint:/var/lib/kubelet/pods/8b36189e-0ae0-498e-8bd9-0391d1f3a47c/volumes/kubernetes.io~projected/kube-api-access-cw889 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c417afb-ca2e-4706-8004-f6651dd11f16/volumes/kubernetes.io~projected/kube-api-access-txxh9:{mountpoint:/var/lib/kubelet/pods/8c417afb-ca2e-4706-8004-f6651dd11f16/volumes/kubernetes.io~projected/kube-api-access-txxh9 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~projected/kube-api-access-k86rq:{mountpoint:/var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~projected/kube-api-access-k86rq major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~projected/kube-api-access-crmlw:{mountpoint:/var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~projected/kube-api-access-crmlw major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acb8307e-d525-404e-ac04-ec75650aa4f2/volumes/kubernetes.io~projected/kube-api-access-jbgpj:{mountpoint:/var/lib/kubelet/pods/acb8307e-d525-404e-ac04-ec75650aa4f2/volumes/kubernetes.io~projected/kube-api-access-jbgpj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7ee20f-a7f2-4994-abea-4c233d8b8da3/volumes/kubernetes.io~projected/kube-api-access-g2gll:{mountpoint:/var/lib/kubelet/pods/bb7ee20f-a7f2-4994-abea-4c233d8b8da3/volumes/kubernetes.io~projected/kube-api-access-g2gll major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdccbf5a-017a-46f7-a82e-df13cc459a30/volumes/kubernetes.io~projected/kube-api-access-ftb8j:{mountpoint:/var/lib/kubelet/pods/bdccbf5a-017a-46f7-a82e-df13cc459a30/volumes/kubernetes.io~projected/kube-api-access-ftb8j major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4119de0-3ce3-4040-8ed3-a3c1366485da/volumes/kubernetes.io~projected/kube-api-access-dw7k7:{mountpoint:/var/lib/kubelet/pods/c4119de0-3ce3-4040-8ed3-a3c1366485da/volumes/kubernetes.io~projected/kube-api-access-dw7k7 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87d2cac-3415-4c41-9ba2-de3dd8ee236f/volumes/kubernetes.io~projected/kube-api-access-vbd89:{mountpoint:/var/lib/kubelet/pods/c87d2cac-3415-4c41-9ba2-de3dd8ee236f/volumes/kubernetes.io~projected/kube-api-access-vbd89 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a136f8-4c59-458c-b2bd-c1645cb9eb28/volumes/kubernetes.io~projected/kube-api-access-d2jh9:{mountpoint:/var/lib/kubelet/pods/d3a136f8-4c59-458c-b2bd-c1645cb9eb28/volumes/kubernetes.io~projected/kube-api-access-d2jh9 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~projected/kube-api-access-qb455:{mountpoint:/var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~projected/kube-api-access-qb455 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~projected/kube-api-access-gj2bv:{mountpoint:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~projected/kube-api-access-gj2bv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e306cc55-def6-43b0-99d2-ea6549ea2a1c/volumes/kubernetes.io~projected/kube-api-access-766w5:{mountpoint:/var/lib/kubelet/pods/e306cc55-def6-43b0-99d2-ea6549ea2a1c/volumes/kubernetes.io~projected/kube-api-access-766w5 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~projected/kube-api-access-hg9hh:{mountpoint:/var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~projected/kube-api-access-hg9hh major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7666d1a-d160-4cc8-a277-208408706b6d/volumes/kubernetes.io~projected/kube-api-access-k5p78:{mountpoint:/var/lib/kubelet/pods/f7666d1a-d160-4cc8-a277-208408706b6d/volumes/kubernetes.io~projected/kube-api-access-k5p78 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/kube-api-access-rfbz2:{mountpoint:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/kube-api-access-rfbz2 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:224 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/62cee506bbead426112680821032ff470fa03c85ad3031ea0506e64add43af40/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/910f21d09a1fcedd04d882c80c3f9648081516631d7823f03d0e41ac61dc9ba8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/938e9b26ff806af33d3888c0601ee1223caa05c670ee67b49ea7f512fc525cdd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/16a91c42104db2e7aabc53591960340eff93c1d7bb02820ad6564b6bfded5380/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d086827d032dae911d7cb0d70a514d62c3559ce9b789106d996f981f4f3f45f1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5d4bbd3e68cba3049368ac0fb766fb8cf8511e2e6875e422c5488890f25bc2d3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3f2ec590d06ad74c23d3802cf3f905c79bd235939a575104fbde961456f11a7a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b43ee69fbbabe9076c3db1af0df9f9d6a24c36d07bdfcfbd7339397d8f424dac/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a44dee7e572059acbd9ba9b0fb12c2b09e44b731ed5902366fb6b97ec77c7028/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/818d12879cb5263e592b5a9efe597fa7b11a2b850d4d37b35f065b6afaa6d0d2/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/5f17fd941e449d376ecaad9c889cd50bd12db33a73ab3d4d11633a1879b97cde/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/ff92b1ffff7340e7f6c195519dcb63b08e39a51fc405774f38020135036f57f0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/551dd38fc33f19d8469b22e7ad2660cad09f553695bdf30b9b8be97f564f2745/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e213ee79dc3e98901245bb9fe2aabde85afcc67c836b9332139d6128b33a2f8b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/4188096f43b8b054a0150e77b099468492826cd2492f65ebb1f1139f1895fcef/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/47e1dd0cdb8d28b77fe20763b418a5aaad0e4a5e2f0d5572cd0e8710ac7df4bb/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4805918f681277f79bf0616a31802e9ce24064feb1bcaebe9f0810791109145a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5c0d0e10819f187ad76746bed129f4f20ca65a051c8337f36a7d7194a647d951/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/200d0e27961d46a65935c26bf3ed2039f12839c0fff7801dd8c6fba751347c37/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e784d608c939560b26702f9173b904b8921fbfffadfc7beb45cc21f07f1395c2/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0c3fefea03299dc259b9020f60c12e25a4bc5d930001a368005d6372f1047ade/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/58018f17fc87245ff7803eedee8581d340026eba485ee70fbbbcde431db6bfba/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fe12f5b11701e778e17ccd5c7b7378c2916788f1709dbeb2b3ba74e1fda475dc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/e02eea5bfb836761efc0d86dd1e5388505ac164d6de419fe8c026f7b9e851202/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5ca9aaf73408da1d8335d9ffa87280e69c013c22ad8a65158515054510c1d177/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/bcf8c939a7973cf2b323cec3f055e9bcc79ecb7e8057006a7da277b4d920b0be/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c12daf07177c83f35f1b9566d5b0863f812f2ce987ae202aa79581b686e73cab/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8eee8a179b83d2fdf22da1c3c48f7f023239c7479dfeb22e6186610d740aa4fd/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c4c0325aebb4eae322db71309b607fbf18c4fc06bd374f90f8a3a6e0a805f0ea/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/01f9dd13772b677d817ca926f264b45debb1acedcdff979961b5c333b76e7bce/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/489cf3b025df7ceaca8f5ac2fea705a3941f3d01171e0e568b746a200db0be14/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/107d82eee82f23da3fabcfb509b59efe425f546768b792bcc38fd6a5ff06070b/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/524eb48fb53bb2fb039fd74ba7c0ee1250ecf689838290ab03f36d0240807180/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/996e9a5db5bdecf07e6525df093e64a7d8f00718bc2a97b09de7e2e6a127e308/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/d7d0a9234922848e8fc0c0644d3dca5729c83e68673bdab0f9ea765947c0d888/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/1c43997396e1426274e07a2e31513ff1f49b0645db98ea2f6621ccf99f3b4913/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/841c9d596b6d01a2b6d0429031aae05bed0e2ca52d66411398604506d3f755a1/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/fb0a3d831436a9a6bf3cd7451cffd51b2def01987c2261ddadbaeb4f5f10f570/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/cd9ae022ff37c1a6399c7ad668f8d7421d334ee3d584ccb348203ef1f14b44d4/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e3c381432ad31dd801de32306c1a07157e751af317a427180a4460e82f0cb378/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/9207797474c05e065c9e28d110354f607585ac969422c288afb653e57ef529be/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e255488547749eb9f28f3db52cf134ab53c6098872365c8eacdcede81635a233/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/23db83cc8b14a26d6934ffc93b16e8ba3018f585b4277d9d33abc978b573bc93/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/181aade976f779fd418a4e200521b4aa2fce097f9947f807ddf71092d684fe40/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/3de5856a9a59bd679e69c9aac2aac2d1aee80fcc585be7dceab38a8ff9368c1c/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/b0c7e9e26e7e0ad77fe55d3844f04c93d612f948093564482a93131102c3ce62/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/dbdc03a021ac1bfca4ab0cd4873ed519384d12686e55d550dacd02e43ff0ae91/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/0ff9eeef374296b2600fc4895480b62e877d8c37def28e15e621ea76888b3e5e/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-46:{mountpoint:/var/lib/containers/storage/overlay/e8899fb98bdaa3f3ca41965e7da70f79672c0d671e0363fa7dbab9afa90f4ecb/merged major:0 minor:46 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/f646d07d24c6fc1314fe6fc055138b724f9968380bf0e6230b609a56a099abba/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ad6acf58a398b73439cddfef6b413b84cda0b472c9cd58a54f32abafbd2451cb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/99d7a5c99f4d1cbc0edda4a4f7d4378a42ab5aa3f9d9f13a6ecfceb95322b4b4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/153520e433aa5722d79b7bd8b8d66d3ce0ad4839c1023c1a034a2d5f3f176cf1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ae8e0105a36e971adad4c8fe4bcc801a269fc64903180ae040e1fbfb8c455324/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ef35cc679f4a37dced010f6cf668c5866823a5e90014d372cbe8c9ff2511f4a8/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1224df89d84da6afaf7017e220514b970a4173e34d44281cabb2ae7808d44520/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/260ac96aade29c7be0a5242573b06d623ebf8ee3df8eccb870eb8125f4d8a32b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3e573a6c6ec0c1be5ad9b1b5eb813980e22c4d4dbaf712c946d00c30189a1a73/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6b92f96d0224632ab4983f12555a9aab486da6062ec5a6c8bc8faed17df46ae0/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 18:46:03.600018 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.599653 4789 manager.go:217] Machine: {Timestamp:2026-10-30 18:46:03.599133452 +0000 UTC m=+0.101043930 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099902 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2df66cacd9ae187d152617b2ce3a40 SystemUUID:ec2df66c-acd9-ae18-7d15-2617b2ce3a40 BootID:170d6af1-d498-47f4-a705-7768f04adc63 Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7666d1a-d160-4cc8-a277-208408706b6d/volumes/kubernetes.io~projected/kube-api-access-k5p78 DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f35a4350246ccaaebb4ca1d6c39a7afcb8f6949ad30265d0eca6df4ab6f2de79/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a00dd2e7174be94eef19f8c98a180f6c5cb08f78e41f7967ee5194952c3b33e7/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48d1db69-50e4-4cc9-a744-2ab11bd3f519/volumes/kubernetes.io~projected/kube-api-access-2lfz9 DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-46 DeviceMajor:0 DeviceMinor:46 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/150b998b-9608-4846-ac54-8a8ec8a7c620/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e306cc55-def6-43b0-99d2-ea6549ea2a1c/volumes/kubernetes.io~projected/kube-api-access-766w5 DeviceMajor:0 DeviceMinor:240 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdccbf5a-017a-46f7-a82e-df13cc459a30/volumes/kubernetes.io~projected/kube-api-access-ftb8j DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~projected/kube-api-access-sljzq DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea62417ad8a443cfebe994240cc15c7c143327c8ae15b463c47fe0d6e62bc95d/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd03fec8a2d73bc268521c19a179c035522019e9589021018daca3e8f608481e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0faaaae953ca8327cbd7112f4a13095268c44ca373ccc7f9e680dbe489789ab7/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/764e2c2641d6419aa4d88057ca7a2bce2feb235970bb75ff1b66b3195b645680/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8b36189e-0ae0-498e-8bd9-0391d1f3a47c/volumes/kubernetes.io~projected/kube-api-access-cw889 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/kube-api-access-dthhg DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~projected/kube-api-access-l4q5j DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/476c9a42e93ceebafc7ada295775e53c20553f90e604da16f7de83fd831a899e/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~projected/kube-api-access-qb455 DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7ee20f-a7f2-4994-abea-4c233d8b8da3/volumes/kubernetes.io~projected/kube-api-access-g2gll DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47c4663a6bd7f8a73ae7d930a3e7f3c4048387775284d3f174e72d32d9a4e9bb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~projected/kube-api-access-crmlw DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfb7dafa6a87debbe2a3049d4c124aa3a5db9ec7b9c018489523981cb5bb3b0e/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f732cf0-c45a-4146-9d07-63c3e1e15ba6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c4304e9a1472ebfd757da5d2813e1ed976439f6f305cb23d2f64e2644b4596f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46806ffc8d15b62c2fd78fe9db10dbb71f10b274832a10960c437eff87f73a35/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a136f8-4c59-458c-b2bd-c1645cb9eb28/volumes/kubernetes.io~projected/kube-api-access-d2jh9 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7cee29d4-8788-4842-b8c5-a2f6e6f3996a/volumes/kubernetes.io~projected/kube-api-access-2g6px DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~projected/kube-api-access-v89cg DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76811b23cac9632e7e9fa85793d747bd73ebb6dfdee5325ed50ef0e4b23ef199/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f72f4fdb7618a788f59ab9368dbc160378dd15ab13522fafdf2fe9234f9006f7/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e294837611b47e13c3c99f5c1f749aff20bdf5c5f4a9690fe6021110490bb8bc/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38e48c7c-d60b-4daf-aa1a-b4578e3d910f/volumes/kubernetes.io~projected/kube-api-access-stdln DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea83faa290b8e647f468748c55b8bc9d3b649952924012c133946e80f7a7cd7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41bb2e81-6a7b-434e-a6c4-184beef52d40/volumes/kubernetes.io~projected/kube-api-access-pkv2k DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/572225d697ad3f781a32c9eef4e751956d7d82aa1850ea82e7f1a3894f666b31/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24b20b3a-54d7-4dcd-b353-f9e28b7654c6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46f2c55b4a83727cf9266f406d0eba5f33e2b162a78751ff7f576920958a34e1/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/837216d426e784b6af4366566b40a50c39d47209b1b5cb1d7e38fe64e6a4de4b/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c87d2cac-3415-4c41-9ba2-de3dd8ee236f/volumes/kubernetes.io~projected/kube-api-access-vbd89 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e097e434daed8f6a0663f6fb67b375dfba0863f6daf5e2c226848aade5906af4/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~projected/kube-api-access-gc4tm DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/361aa627-cb6e-4b51-96b4-f66cc3ebb23a/volumes/kubernetes.io~projected/kube-api-access-zq882 DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~projected/kube-api-access-gj2bv DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/411447d733b4b011b8adc0267b4ee6fbaec56299355cea07d0e76a4611b97cbd/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94f91f0e-7db9-426d-ac87-4c055d52abaa/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7fd86c83-2fdf-408f-a2f3-507b654c1f45/volumes/kubernetes.io~projected/kube-api-access-md79n DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8db9995d-8178-4095-88a2-01ca522016d2/volumes/kubernetes.io~projected/kube-api-access-k86rq DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7631dac7cca660db7e966ba5a02e12f2c37bab714a257a622c9b9b093001511/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f076db4-f03f-4bf3-be39-e899370f953c/volumes/kubernetes.io~projected/kube-api-access-pzcm7 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/497867c2-a03d-4717-b845-1d0ef71495d5/volumes/kubernetes.io~projected/kube-api-access-t99zj DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7b17feeac5c8cdb9404db2d78d401ff42762ce85ce97a99a0c39d75c9085dd8/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d3eacfd-ea78-48c0-8968-da01c34ec561/volumes/kubernetes.io~projected/kube-api-access-hrnhg DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae3f1113b221c060db8b8f7f7be662d173f450d4d0c987ff353a84240dcb78d1/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7224a54e-ce8c-4b82-9247-11376266019b/volumes/kubernetes.io~projected/kube-api-access-nnvt7 DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc/volumes/kubernetes.io~projected/kube-api-access-hg9hh DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0605710c-1a4a-47f2-a7bc-4079a97e9db4/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c417afb-ca2e-4706-8004-f6651dd11f16/volumes/kubernetes.io~projected/kube-api-access-txxh9 DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes/kubernetes.io~projected/kube-api-access-rfbz2 DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4119de0-3ce3-4040-8ed3-a3c1366485da/volumes/kubernetes.io~projected/kube-api-access-dw7k7 DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/de998501-b81d-4fa5-b100-34ee068f8b03/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acb8307e-d525-404e-ac04-ec75650aa4f2/volumes/kubernetes.io~projected/kube-api-access-jbgpj DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0faaaae953ca832 MacAddress:8e:b2:b1:55:6d:1c Speed:10000 Mtu:8901} {Name:411447d733b4b01 MacAddress:72:b5:61:53:b6:58 Speed:10000 Mtu:8901} {Name:46f2c55b4a83727 MacAddress:b6:67:e9:d2:1d:b4 Speed:10000 Mtu:8901} {Name:476c9a42e93ceeb MacAddress:ca:62:9e:12:8f:a8 Speed:10000 Mtu:8901} {Name:6b18f098a9e8902 MacAddress:62:b3:21:47:17:44 Speed:10000 Mtu:8901} {Name:764e2c2641d6419 MacAddress:da:78:30:9d:2d:b4 Speed:10000 Mtu:8901} {Name:76811b23cac9632 MacAddress:d2:16:28:e9:41:8e Speed:10000 Mtu:8901} {Name:837216d426e784b MacAddress:6a:ab:af:c2:89:f2 Speed:10000 Mtu:8901} {Name:a00dd2e7174be94 MacAddress:72:d0:91:56:63:04 Speed:10000 Mtu:8901} {Name:b7631dac7cca660 MacAddress:7a:01:77:a8:20:6e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d0:74:b9:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:3b:ae:fa:eb:d4 Speed:0 Mtu:8901} {Name:c7b17feeac5c8cd MacAddress:26:93:e7:dd:3c:da Speed:10000 Mtu:8901} {Name:dfb7dafa6a87deb MacAddress:c6:9e:fd:a8:28:c0 Speed:10000 Mtu:8901} {Name:e097e434daed8f6 MacAddress:6a:ea:62:db:1c:49 Speed:10000 Mtu:8901} {Name:ea62417ad8a443c MacAddress:fe:bf:32:91:75:28 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d0:74:b9:11 Speed:0 Mtu:9001} {Name:f35a4350246ccaa MacAddress:52:b7:c6:94:0c:21 Speed:10000 Mtu:8901} {Name:f72f4fdb7618a78 MacAddress:4a:a3:9c:71:54:b2 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:7b:60:8e:fb:ed Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:6d:df:18:bf:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:46:03.600198 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.600021 4789 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:46:03.600198 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.600125 4789 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:46:03.601093 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601047 4789 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:46:03.601595 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601122 4789 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-46.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:46:03.601675 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601599 4789 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:46:03.601675 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601612 4789 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:46:03.601675 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601632 4789 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:46:03.601675 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601649 4789 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:46:03.601797 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.601735 4789 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:46:03.602311 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602296 4789 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:46:03.602426 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602416 4789 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:46:03.602463 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602453 4789 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:46:03.602497 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602472 4789 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:46:03.602497 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602484 4789 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:46:03.602497 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602492 4789 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:46:03.602599 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.602500 4789 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:46:03.603497 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.603479 4789 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:46:03.603497 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.603498 4789 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:46:03.603963 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.603947 4789 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:46:03.618644 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.618625 4789 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:46:03.618644 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.618643 4789 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:46:03.618766 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.618750 4789 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:46:03.618947 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.618935 4789 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:46:03.618981 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.618965 4789 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:46:03.619078 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619069 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:46:03.619114 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619084 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:46:03.619114 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619090 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:46:03.619114 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619095 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:46:03.619114 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619102 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:46:03.619114 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619111 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619120 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619126 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619133 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619138 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619147 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619155 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619180 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:46:03.619284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.619187 4789 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:46:03.622761 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.622748 4789 server.go:1263] "Started kubelet" Oct 30 18:46:03.624095 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.622839 4789 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:46:03.624095 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.622879 4789 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:46:03.624095 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.622985 4789 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:46:03.623349 ip-10-0-136-46 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:46:03.624724 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.624705 4789 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:46:03.624940 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.624894 4789 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:46:03.630136 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630081 4789 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:46:03.630136 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630114 4789 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630164 4789 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:40:22 +0000 UTC" deadline="2028-06-27 12:45:39.435343916 +0000 UTC" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630182 4789 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14537h59m35.805164179s" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630200 4789 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630231 4789 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.630203 4789 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.630593 4789 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632165 4789 factory.go:55] Registering systemd factory Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632187 4789 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632586 4789 factory.go:153] Registering CRI-O factory Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632598 4789 factory.go:223] Registration of the crio container factory successfully Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632637 4789 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632652 4789 factory.go:103] Registering Raw factory Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.632661 4789 manager.go:1199] Started watching for new ooms in manager Oct 30 18:46:03.633399 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.633097 4789 manager.go:319] Starting recovery of all containers Oct 30 18:46:03.634335 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634309 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" volumeName="kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy" seLinuxMountContext="" Oct 30 18:46:03.634335 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634336 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f732cf0-c45a-4146-9d07-63c3e1e15ba6" volumeName="kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634345 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634361 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666d1a-d160-4cc8-a277-208408706b6d" volumeName="kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634371 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634380 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" volumeName="kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634388 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41bb2e81-6a7b-434e-a6c4-184beef52d40" volumeName="kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634396 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634404 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87d2cac-3415-4c41-9ba2-de3dd8ee236f" volumeName="kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634411 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg" seLinuxMountContext="" Oct 30 18:46:03.634420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634419 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b36189e-0ae0-498e-8bd9-0391d1f3a47c" volumeName="kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634426 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cdd95a5-32ae-4c7a-a067-bc40ad4bc826" volumeName="kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634433 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4119de0-3ce3-4040-8ed3-a3c1366485da" volumeName="kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634440 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a136f8-4c59-458c-b2bd-c1645cb9eb28" volumeName="kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634456 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de998501-b81d-4fa5-b100-34ee068f8b03" volumeName="kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634466 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e306cc55-def6-43b0-99d2-ea6549ea2a1c" volumeName="kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634474 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0605710c-1a4a-47f2-a7bc-4079a97e9db4" volumeName="kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634491 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" volumeName="kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634499 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="497867c2-a03d-4717-b845-1d0ef71495d5" volumeName="kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634508 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db9995d-8178-4095-88a2-01ca522016d2" volumeName="kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634515 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb8307e-d525-404e-ac04-ec75650aa4f2" volumeName="kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634521 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de998501-b81d-4fa5-b100-34ee068f8b03" volumeName="kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634528 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634535 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" volumeName="kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634541 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94f91f0e-7db9-426d-ac87-4c055d52abaa" volumeName="kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634548 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a136f8-4c59-458c-b2bd-c1645cb9eb28" volumeName="kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634555 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb8307e-d525-404e-ac04-ec75650aa4f2" volumeName="kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634562 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0605710c-1a4a-47f2-a7bc-4079a97e9db4" volumeName="kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634586 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f076db4-f03f-4bf3-be39-e899370f953c" volumeName="kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634594 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6" volumeName="kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634601 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd86c83-2fdf-408f-a2f3-507b654c1f45" volumeName="kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634609 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db9995d-8178-4095-88a2-01ca522016d2" volumeName="kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634619 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440" volumeName="kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634628 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634638 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f732cf0-c45a-4146-9d07-63c3e1e15ba6" volumeName="kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634645 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates" seLinuxMountContext="" Oct 30 18:46:03.634669 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634652 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb8307e-d525-404e-ac04-ec75650aa4f2" volumeName="kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634661 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634698 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634707 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634715 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634722 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3eacfd-ea78-48c0-8968-da01c34ec561" volumeName="kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634729 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d3eacfd-ea78-48c0-8968-da01c34ec561" volumeName="kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634736 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b20b3a-54d7-4dcd-b353-f9e28b7654c6" volumeName="kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634743 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6" volumeName="kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634750 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b20b3a-54d7-4dcd-b353-f9e28b7654c6" volumeName="kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634758 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634765 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7224a54e-ce8c-4b82-9247-11376266019b" volumeName="kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634772 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c417afb-ca2e-4706-8004-f6651dd11f16" volumeName="kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634782 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94f91f0e-7db9-426d-ac87-4c055d52abaa" volumeName="kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634789 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7ee20f-a7f2-4994-abea-4c233d8b8da3" volumeName="kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634795 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" volumeName="kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634803 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e48c7c-d60b-4daf-aa1a-b4578e3d910f" volumeName="kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634810 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634817 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdccbf5a-017a-46f7-a82e-df13cc459a30" volumeName="kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634824 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f732cf0-c45a-4146-9d07-63c3e1e15ba6" volumeName="kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634849 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24b20b3a-54d7-4dcd-b353-f9e28b7654c6" volumeName="kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634858 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38e48c7c-d60b-4daf-aa1a-b4578e3d910f" volumeName="kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634865 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" volumeName="kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634872 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" volumeName="kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634879 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" volumeName="kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634886 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fd86c83-2fdf-408f-a2f3-507b654c1f45" volumeName="kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634894 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8db9995d-8178-4095-88a2-01ca522016d2" volumeName="kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634902 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b998b-9608-4846-ac54-8a8ec8a7c620" volumeName="kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634913 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94f91f0e-7db9-426d-ac87-4c055d52abaa" volumeName="kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634920 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de998501-b81d-4fa5-b100-34ee068f8b03" volumeName="kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634927 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de998501-b81d-4fa5-b100-34ee068f8b03" volumeName="kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634937 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de998501-b81d-4fa5-b100-34ee068f8b03" volumeName="kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634944 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634952 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634961 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0605710c-1a4a-47f2-a7bc-4079a97e9db4" volumeName="kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634967 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="150b998b-9608-4846-ac54-8a8ec8a7c620" volumeName="kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634974 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d1db69-50e4-4cc9-a744-2ab11bd3f519" volumeName="kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634982 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6" volumeName="kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634989 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528cf1e4-dafb-406c-a37b-c055df14a5e1" volumeName="kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.634997 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440" volumeName="kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635005 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c6b33a9-100f-42a4-b231-8e00e3d3520b" volumeName="kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635012 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94f91f0e-7db9-426d-ac87-4c055d52abaa" volumeName="kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635019 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87d2cac-3415-4c41-9ba2-de3dd8ee236f" volumeName="kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635025 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e306cc55-def6-43b0-99d2-ea6549ea2a1c" volumeName="kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635032 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7666d1a-d160-4cc8-a277-208408706b6d" volumeName="kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635039 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e306cc55-def6-43b0-99d2-ea6549ea2a1c" volumeName="kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635046 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635054 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635063 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" volumeName="kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635070 4789 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcb96ed0-2dfb-449a-8389-095c46386e9b" volumeName="kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2" seLinuxMountContext="" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635079 4789 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:46:03.635477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.635084 4789 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:46:03.639572 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.639555 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:03.642624 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.642578 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:03.658973 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.658845 4789 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:46:03.660122 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.660104 4789 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:46:03.660299 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.660128 4789 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:46:03.660299 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.660149 4789 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:46:03.660299 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.660187 4789 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.694927 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="52d537218be09499b353d61b82b34082050f82b480f588dfe50b33e916ca4f5b" exitCode=0 Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.694961 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="8e56309d3ec2c3a24ca0cf327f9de01d0eed76f22054a9b3f8024f958ca38389" exitCode=0 Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.694973 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="78e723517c5e25b3b8783c4f218427684b10f0273253b1939eaae40118305fdf" exitCode=0 Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.694985 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="8c10ab35d7d168719fee3535e6ca18768883bc557f7ed0b41752d70ad87132a4" exitCode=0 Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.694995 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="0825d0db7e306c4df27214ed68643bd5829b881f6fbec70b2b79c8e04e0720cf" exitCode=0 Oct 30 18:46:03.695010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.695007 4789 generic.go:356] "Generic (PLEG): container finished" podID="7cee29d4-8788-4842-b8c5-a2f6e6f3996a" containerID="46027b51264cd72c50a9ba2ebe28a78429cdaad053967156a35b9374a3a693de" exitCode=0 Oct 30 18:46:03.705404 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.705380 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:03.705517 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.705415 4789 generic.go:356] "Generic (PLEG): container finished" podID="94f91f0e-7db9-426d-ac87-4c055d52abaa" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" exitCode=255 Oct 30 18:46:03.709513 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.709493 4789 generic.go:356] "Generic (PLEG): container finished" podID="e611e756739ad427bda17a40f0e6a03c" containerID="a266f0efa4b741781d52d922ed01bb177f2f913fd8f852cf3a29fa045e4e37d5" exitCode=0 Oct 30 18:46:03.726493 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.726477 4789 manager.go:324] Recovery completed Oct 30 18:46:03.730681 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.730666 4789 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:46:03.743903 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.743886 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.745451 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.745437 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.745507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.745465 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.745507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.745475 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.747687 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747669 4789 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:46:03.747687 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747683 4789 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:46:03.747687 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747687 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.747852 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747702 4789 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:46:03.747889 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747868 4789 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:46:03.747921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747879 4789 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:46:03.747921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747895 4789 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:46:03.747921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747900 4789 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:46:03.747921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747906 4789 policy_none.go:50] "Start" Oct 30 18:46:03.747921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747915 4789 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:46:03.748117 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.747925 4789 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:46:03.748117 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748049 4789 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:46:03.748117 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748056 4789 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:46:03.748117 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748062 4789 policy_none.go:44] "Start" Oct 30 18:46:03.748716 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748701 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.748784 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748732 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.748784 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.748747 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.752980 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.752955 4789 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:46:03.755652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.755636 4789 manager.go:341] "Starting Device Plugin manager" Oct 30 18:46:03.755764 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.755665 4789 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:46:03.755764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.755670 4789 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:46:03.755764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.755676 4789 server.go:87] "Starting device plugin registration server" Oct 30 18:46:03.755949 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.755935 4789 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:46:03.756629 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.755952 4789 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:46:03.756717 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.756656 4789 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:46:03.756818 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.756783 4789 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:46:03.756893 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.756831 4789 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:46:03.756893 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.756838 4789 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:46:03.757003 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.756888 4789 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:46:03.756815366Z","UUID":"c42086ed-4dca-40ee-95e0-4a951cf17e0f","Handler":null,"Name":"","Endpoint":""} Oct 30 18:46:03.758462 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.758445 4789 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:46:03.758554 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.758470 4789 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:46:03.759910 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.759892 4789 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:46:03.759995 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.759929 4789 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" Oct 30 18:46:03.760437 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.760396 4789 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal"] Oct 30 18:46:03.760562 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.760549 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.761501 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.761483 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.761585 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.761515 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.761585 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.761529 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.761888 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.761870 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.762025 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762010 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.762847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762830 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.762942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762857 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.762942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762870 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.762942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762888 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.762942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762911 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.762942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.762926 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.763182 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.763031 4789 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.763451 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763409 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerStarted","Data":"3e01f554dedd40dc520368dcb044899abbfd0f237560391b9ca31b7815488c49"} Oct 30 18:46:03.763536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763452 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerDied","Data":"a266f0efa4b741781d52d922ed01bb177f2f913fd8f852cf3a29fa045e4e37d5"} Oct 30 18:46:03.763536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763460 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.763536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763472 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"e611e756739ad427bda17a40f0e6a03c","Type":"ContainerStarted","Data":"16ffec36d13d4aeb4b2335b4670374367acf39a2938304ada2d65dc1f60cedd3"} Oct 30 18:46:03.763536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763491 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"29dce357bba57b17e7ff4a65afbed63d","Type":"ContainerStarted","Data":"f48df332f7dffb05e74511fb5932ff55269f607f721d8e4f7462bceb9482f792"} Oct 30 18:46:03.763536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.763508 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" event={"ID":"29dce357bba57b17e7ff4a65afbed63d","Type":"ContainerStarted","Data":"34be2616b8f9f75a665d89e19942f2e6922127c5f343edc285225432da2b882d"} Oct 30 18:46:03.764380 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.764362 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.764448 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.764388 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.764448 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.764404 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.764547 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.764534 4789 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-46.us-west-2.compute.internal\" not found" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.800970 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.800945 4789 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:46:03.801113 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.801098 4789 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:46:03.802062 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.802048 4789 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:46:03.831328 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.831306 4789 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.850177 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.850157 4789 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:46:03.857478 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.857459 4789 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:46:03.858406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.858385 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:46:03.858468 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.858411 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:46:03.858468 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.858422 4789 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-46.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:46:03.858468 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.858460 4789 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.889390 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.889369 4789 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:46:03.889458 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.889409 4789 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.889458 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.889419 4789 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.901333 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.901314 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.901391 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.901337 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.901391 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.901356 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.946164 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.946120 4789 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:46:03.946164 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:03.946150 4789 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.967464 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.967447 4789 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:03.967513 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:03.967499 4789 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002041 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002020 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002079 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002102 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/29dce357bba57b17e7ff4a65afbed63d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"29dce357bba57b17e7ff4a65afbed63d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002177 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002123 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002177 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002143 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.002177 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.002173 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e611e756739ad427bda17a40f0e6a03c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" (UID: \"e611e756739ad427bda17a40f0e6a03c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.602667 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.602638 4789 apiserver.go:52] "Watching apiserver" Oct 30 18:46:04.714855 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.714827 4789 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.715001 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.714972 4789 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.745875 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.745849 4789 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:46:04.746618 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.746596 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xw9bs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-nlzxq","openshift-image-registry/image-registry-6f85f87669-rmcbh","openshift-ovn-kubernetes/ovnkube-node-djnd7","openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw","openshift-service-ca/service-ca-5fffc97bdf-8npkv","openshift-multus/multus-pskdl","kube-system/kas-connection-checker-685895c7c9-54zh4","openshift-insights/periodic-gathering-9fgqw-rvvkn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k","openshift-multus/multus-additional-cni-plugins-qlzjf","openshift-network-diagnostics/network-check-target-lbk7g","openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx","openshift-network-operator/iptables-alerter-zsxjb","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc","openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv","kube-system/konnectivity-agent-rs2ft","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw","kube-system/global-pull-secret-syncer-vnc44","openshift-dns/dns-default-q7r4n","openshift-image-registry/image-registry-bbc68679b-ln4qc","openshift-insights/insights-runtime-extractor-2ckq9","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m","kube-system/kas-connection-checker-685895c7c9-bzdtx","kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal","openshift-dns/node-resolver-s65mr","openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht","openshift-ingress/router-default-5978b965c7-kbthb","openshift-network-console/networking-console-plugin-86c478bb88-jnbnw","kube-system/kas-connection-checker-685895c7c9-fj8mt","openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w","openshift-cluster-node-tuning-operator/tuned-5xbl5","openshift-insights/insights-operator-7c4986d449-95sk7","openshift-multus/network-metrics-daemon-jgml5","openshift-image-registry/node-ca-4cm6p"] Oct 30 18:46:04.747905 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.747887 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.748185 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.748166 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.748551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.748533 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.749136 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.749117 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:04.753942 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.753921 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:04.755849 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.755831 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:04.759563 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.759543 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.760167 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.760156 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.761534 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.761519 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.763695 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.763676 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.771991 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.771972 4789 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:46:04.772063 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:04.772008 4789 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.772525 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.772511 4789 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:46:04.772564 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:04.772552 4789 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-46.us-west-2.compute.internal" Oct 30 18:46:04.804597 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804560 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.804859 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804842 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.804934 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804885 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.804934 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804917 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.805020 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804967 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805020 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.804976 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805020 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805001 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805131 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805022 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.805194 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805141 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.805194 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805143 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:46:04.805194 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805170 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 18:46:04.805194 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805170 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 18:46:04.805351 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805194 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.805351 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805233 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805351 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805303 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805351 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805318 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.805478 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805469 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 18:46:04.805575 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805538 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.805916 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805901 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 18:46:04.805999 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805902 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 18:46:04.805999 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.805982 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kzsm2\"" Oct 30 18:46:04.806375 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.806361 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 18:46:04.806747 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.806728 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6fg8\"" Oct 30 18:46:04.806860 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.806775 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 18:46:04.806860 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.806811 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 18:46:04.818320 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.818300 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 18:46:04.836601 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.836583 4789 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:46:04.837152 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.837138 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r2fz8\"" Oct 30 18:46:04.837255 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.837241 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 18:46:04.845100 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.845084 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 18:46:04.845162 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.845084 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 18:46:04.851065 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.851049 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 18:46:04.851193 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.851092 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ffnnw\"" Oct 30 18:46:04.851193 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.851135 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q6v4b\"" Oct 30 18:46:04.851333 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.851321 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 18:46:04.851580 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.851566 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hbsvt\"" Oct 30 18:46:04.852036 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.852020 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 18:46:04.852115 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.852102 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 18:46:04.899515 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.899457 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:46:04.899515 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.899499 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 18:46:04.900320 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900301 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:46:04.900524 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900499 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:46:04.900596 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900570 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:46:04.900630 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900603 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29qlm\"" Oct 30 18:46:04.900680 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900668 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:46:04.900998 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.900984 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 18:46:04.901030 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901004 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 18:46:04.901068 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901028 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 18:46:04.901098 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901066 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lj6ts\"" Oct 30 18:46:04.901098 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901065 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:46:04.901098 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901068 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hz6tx\"" Oct 30 18:46:04.901387 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901373 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-crxtc\"" Oct 30 18:46:04.901435 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901393 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-68fw6\"" Oct 30 18:46:04.901490 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901436 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k86kp\"" Oct 30 18:46:04.901490 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901448 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vvlg2\"" Oct 30 18:46:04.901490 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901449 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-smsmz\"" Oct 30 18:46:04.901490 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901476 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77vpl\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901492 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4rzs\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901481 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tqgpw\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901515 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vjgl9\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901518 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-56wtl\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901528 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-dfwxh\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901543 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5tsjg\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901549 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n4vwp\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901557 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901564 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-q9tbn\"" Oct 30 18:46:04.901652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901627 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpjwm\"" Oct 30 18:46:04.901915 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901702 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pwfgl\"" Oct 30 18:46:04.901915 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901837 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgxcg\"" Oct 30 18:46:04.901915 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901907 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qqsp5\"" Oct 30 18:46:04.902006 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901933 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r47g8\"" Oct 30 18:46:04.902006 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.901975 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:46:04.903785 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.903769 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:46:04.904024 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904007 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904031 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904008 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904056 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904058 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904113 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904128 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:46:04.904454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904148 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:46:04.904454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904162 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904190 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904396 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904395 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904655 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904469 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904655 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904497 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904655 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904510 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.904822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904670 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:46:04.904822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904711 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:46:04.904822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904798 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904812 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.904822 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904820 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904833 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904836 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904838 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904874 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904874 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.904986 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905000 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905001 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905031 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905037 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905031 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905033 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905064 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905091 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905091 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905153 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905189 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905199 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905228 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905257 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 18:46:04.905626 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905471 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:46:04.905908 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.905894 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 18:46:04.909072 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909057 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.909116 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909080 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.909116 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909099 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:46:04.909222 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909115 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.909222 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909140 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:46:04.909222 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909185 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909239 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909290 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909328 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb455\" (UniqueName: \"kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909352 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909370 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e306cc55-def6-43b0-99d2-ea6549ea2a1c-tmp-dir\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909378 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.909424 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909407 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909430 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909488 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909513 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909521 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909545 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7k7\" (UniqueName: \"kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7\") pod \"network-check-source-6f78d59f99-f6bkw\" (UID: \"c4119de0-3ce3-4040-8ed3-a3c1366485da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909567 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/150b998b-9608-4846-ac54-8a8ec8a7c620-konnectivity-ca\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909573 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/150b998b-9608-4846-ac54-8a8ec8a7c620-agent-certs\") pod \"konnectivity-agent-rs2ft\" (UID: \"150b998b-9608-4846-ac54-8a8ec8a7c620\") " pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909641 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.909715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909709 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909738 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909762 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909788 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909819 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909835 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-iptables-alerter-script\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909843 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909869 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89cg\" (UniqueName: \"kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909955 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.909999 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:04.910042 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910039 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910080 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910155 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2bv\" (UniqueName: \"kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910175 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910193 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910229 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910246 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910283 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9hh\" (UniqueName: \"kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910313 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxh9\" (UniqueName: \"kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9\") pod \"kas-connection-checker-685895c7c9-fj8mt\" (UID: \"8c417afb-ca2e-4706-8004-f6651dd11f16\") " pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910341 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910368 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910316 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-original-pull-secret\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910393 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910434 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910448 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-service-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910462 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.910498 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910506 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910555 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftb8j\" (UniqueName: \"kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j\") pod \"migrator-66df5d496b-g22qv\" (UID: \"bdccbf5a-017a-46f7-a82e-df13cc459a30\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910566 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910573 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-tuned\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910586 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910622 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-cni-binary-copy\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910658 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910693 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910722 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md79n\" (UniqueName: \"kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910739 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86rq\" (UniqueName: \"kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910754 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-tmp\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910758 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910785 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910810 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4tm\" (UniqueName: \"kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910836 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99zj\" (UniqueName: \"kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj\") pod \"kas-connection-checker-685895c7c9-54zh4\" (UID: \"497867c2-a03d-4717-b845-1d0ef71495d5\") " pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910865 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910893 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910916 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910941 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmlw\" (UniqueName: \"kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910965 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.910987 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-default-certificate\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911018 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911034 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911056 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-tmp\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911097 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911181 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-env-overrides\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911230 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljzq\" (UniqueName: \"kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911339 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q5j\" (UniqueName: \"kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.911372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911371 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911403 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911431 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvt7\" (UniqueName: \"kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7\") pod \"volume-data-source-validator-58c9f5f46d-2tt7k\" (UID: \"7224a54e-ce8c-4b82-9247-11376266019b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911435 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-binary-copy\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911459 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911487 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911515 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911540 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911564 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911577 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911604 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5p78\" (UniqueName: \"kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911630 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911655 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911678 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911704 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911714 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911731 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911758 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-trusted-ca\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911752 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8db9995d-8178-4095-88a2-01ca522016d2-signing-cabundle\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911951 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911968 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-serviceca\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.911969 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c6b33a9-100f-42a4-b231-8e00e3d3520b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912016 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912045 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912072 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912124 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912143 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.912148 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912150 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912180 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnhg\" (UniqueName: \"kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912240 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912305 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-stats-auth\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912317 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912355 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912425 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912445 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912466 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912473 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912490 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912506 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912548 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912564 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912581 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912596 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912680 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94f91f0e-7db9-426d-ac87-4c055d52abaa-serving-cert\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912782 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-serving-cert\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912788 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94f91f0e-7db9-426d-ac87-4c055d52abaa-config\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912820 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912837 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912856 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.912886 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912884 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdln\" (UniqueName: \"kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912899 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912914 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912939 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgpj\" (UniqueName: \"kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912950 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.912986 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913007 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913031 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913061 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913086 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3eacfd-ea78-48c0-8968-da01c34ec561-tmp-dir\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913135 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913144 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-snapshots\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913169 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913186 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913243 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913274 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e306cc55-def6-43b0-99d2-ea6549ea2a1c-config-volume\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913276 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913321 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913346 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913370 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913374 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-serving-cert\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913410 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913414 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd86c83-2fdf-408f-a2f3-507b654c1f45-data-volume\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913446 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913465 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913490 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913508 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkv2k\" (UniqueName: \"kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k\") pod \"kas-connection-checker-685895c7c9-bzdtx\" (UID: \"41bb2e81-6a7b-434e-a6c4-184beef52d40\") " pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913534 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913553 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913579 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jh9\" (UniqueName: \"kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913605 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913619 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f7666d1a-d160-4cc8-a277-208408706b6d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913630 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913654 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913688 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8db9995d-8178-4095-88a2-01ca522016d2-signing-key\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913739 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:04.913768 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913762 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913890 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de998501-b81d-4fa5-b100-34ee068f8b03-ovn-node-metrics-cert\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913893 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913929 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6px\" (UniqueName: \"kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913947 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913966 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.913992 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914017 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914047 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914093 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914117 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914140 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914187 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914229 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914240 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914260 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914273 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-cert\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914290 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914321 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfz9\" (UniqueName: \"kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9\") pod \"migrator-66df5d496b-nvg6w\" (UID: \"48d1db69-50e4-4cc9-a744-2ab11bd3f519\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914346 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-script-lib\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914347 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914391 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914422 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914456 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914487 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914509 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914544 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914573 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914577 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-daemon-config\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914616 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de998501-b81d-4fa5-b100-34ee068f8b03-ovnkube-config\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914615 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914647 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd89\" (UniqueName: \"kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914664 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914681 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914697 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914664 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914712 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.914729 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw889\" (UniqueName: \"kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915071 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915157 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcm7\" (UniqueName: \"kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915197 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-config\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915241 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915261 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766w5\" (UniqueName: \"kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.915470 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.915314 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.967867 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.967845 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gll\" (UniqueName: \"kubernetes.io/projected/bb7ee20f-a7f2-4994-abea-4c233d8b8da3-kube-api-access-g2gll\") pod \"network-check-target-lbk7g\" (UID: \"bb7ee20f-a7f2-4994-abea-4c233d8b8da3\") " pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:46:04.969714 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.969694 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89cg\" (UniqueName: \"kubernetes.io/projected/24b20b3a-54d7-4dcd-b353-f9e28b7654c6-kube-api-access-v89cg\") pod \"service-ca-operator-544d749c8c-gsbjx\" (UID: \"24b20b3a-54d7-4dcd-b353-f9e28b7654c6\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" Oct 30 18:46:04.970532 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970510 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2bv\" (UniqueName: \"kubernetes.io/projected/de998501-b81d-4fa5-b100-34ee068f8b03-kube-api-access-gj2bv\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:04.970532 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970522 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkv2k\" (UniqueName: \"kubernetes.io/projected/41bb2e81-6a7b-434e-a6c4-184beef52d40-kube-api-access-pkv2k\") pod \"kas-connection-checker-685895c7c9-bzdtx\" (UID: \"41bb2e81-6a7b-434e-a6c4-184beef52d40\") " pod="kube-system/kas-connection-checker-685895c7c9-bzdtx" Oct 30 18:46:04.970776 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970758 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgpj\" (UniqueName: \"kubernetes.io/projected/acb8307e-d525-404e-ac04-ec75650aa4f2-kube-api-access-jbgpj\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:04.970847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970801 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxh9\" (UniqueName: \"kubernetes.io/projected/8c417afb-ca2e-4706-8004-f6651dd11f16-kube-api-access-txxh9\") pod \"kas-connection-checker-685895c7c9-fj8mt\" (UID: \"8c417afb-ca2e-4706-8004-f6651dd11f16\") " pod="kube-system/kas-connection-checker-685895c7c9-fj8mt" Oct 30 18:46:04.970847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970834 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7k7\" (UniqueName: \"kubernetes.io/projected/c4119de0-3ce3-4040-8ed3-a3c1366485da-kube-api-access-dw7k7\") pod \"network-check-source-6f78d59f99-f6bkw\" (UID: \"c4119de0-3ce3-4040-8ed3-a3c1366485da\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-f6bkw" Oct 30 18:46:04.970921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.970880 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5p78\" (UniqueName: \"kubernetes.io/projected/f7666d1a-d160-4cc8-a277-208408706b6d-kube-api-access-k5p78\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:04.978155 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.978136 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.978389 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.978371 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftb8j\" (UniqueName: \"kubernetes.io/projected/bdccbf5a-017a-46f7-a82e-df13cc459a30-kube-api-access-ftb8j\") pod \"migrator-66df5d496b-g22qv\" (UID: \"bdccbf5a-017a-46f7-a82e-df13cc459a30\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" Oct 30 18:46:04.978542 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.978526 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:04.985117 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.985097 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfz9\" (UniqueName: \"kubernetes.io/projected/48d1db69-50e4-4cc9-a744-2ab11bd3f519-kube-api-access-2lfz9\") pod \"migrator-66df5d496b-nvg6w\" (UID: \"48d1db69-50e4-4cc9-a744-2ab11bd3f519\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" Oct 30 18:46:04.985198 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.985136 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q5j\" (UniqueName: \"kubernetes.io/projected/1f732cf0-c45a-4146-9d07-63c3e1e15ba6-kube-api-access-l4q5j\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m\" (UID: \"1f732cf0-c45a-4146-9d07-63c3e1e15ba6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" Oct 30 18:46:04.985198 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.985156 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcm7\" (UniqueName: \"kubernetes.io/projected/2f076db4-f03f-4bf3-be39-e899370f953c-kube-api-access-pzcm7\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:04.990733 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.990717 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.991431 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.991414 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99zj\" (UniqueName: \"kubernetes.io/projected/497867c2-a03d-4717-b845-1d0ef71495d5-kube-api-access-t99zj\") pod \"kas-connection-checker-685895c7c9-54zh4\" (UID: \"497867c2-a03d-4717-b845-1d0ef71495d5\") " pod="kube-system/kas-connection-checker-685895c7c9-54zh4" Oct 30 18:46:04.991591 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.991574 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"periodic-gathering-9fgqw-rvvkn\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:46:04.991990 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.991973 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdln\" (UniqueName: \"kubernetes.io/projected/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-kube-api-access-stdln\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:04.992188 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.992170 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb455\" (UniqueName: \"kubernetes.io/projected/dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440-kube-api-access-qb455\") pod \"ingress-canary-xw9bs\" (UID: \"dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440\") " pod="openshift-ingress-canary/ingress-canary-xw9bs" Oct 30 18:46:04.992398 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.992381 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6px\" (UniqueName: \"kubernetes.io/projected/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-kube-api-access-2g6px\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:04.992565 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.992551 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86rq\" (UniqueName: \"kubernetes.io/projected/8db9995d-8178-4095-88a2-01ca522016d2-kube-api-access-k86rq\") pod \"service-ca-5fffc97bdf-8npkv\" (UID: \"8db9995d-8178-4095-88a2-01ca522016d2\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:04.992565 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.992557 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:04.995327 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.995313 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766w5\" (UniqueName: \"kubernetes.io/projected/e306cc55-def6-43b0-99d2-ea6549ea2a1c-kube-api-access-766w5\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:04.996536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.996521 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw889\" (UniqueName: \"kubernetes.io/projected/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-kube-api-access-cw889\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:04.997338 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:04.997323 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd89\" (UniqueName: \"kubernetes.io/projected/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-kube-api-access-vbd89\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:05.000241 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000223 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9hh\" (UniqueName: \"kubernetes.io/projected/e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc-kube-api-access-hg9hh\") pod \"insights-operator-7c4986d449-95sk7\" (UID: \"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc\") " pod="openshift-insights/insights-operator-7c4986d449-95sk7" Oct 30 18:46:05.000311 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000249 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmlw\" (UniqueName: \"kubernetes.io/projected/94f91f0e-7db9-426d-ac87-4c055d52abaa-kube-api-access-crmlw\") pod \"console-operator-7cd56b8dbc-nlzxq\" (UID: \"94f91f0e-7db9-426d-ac87-4c055d52abaa\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:05.000311 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000253 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md79n\" (UniqueName: \"kubernetes.io/projected/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-api-access-md79n\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.000311 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000293 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljzq\" (UniqueName: \"kubernetes.io/projected/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-kube-api-access-sljzq\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.000408 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000317 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnhg\" (UniqueName: \"kubernetes.io/projected/0d3eacfd-ea78-48c0-8968-da01c34ec561-kube-api-access-hrnhg\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:05.000456 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.000441 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4tm\" (UniqueName: \"kubernetes.io/projected/0605710c-1a4a-47f2-a7bc-4079a97e9db4-kube-api-access-gc4tm\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:05.001336 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.001320 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvt7\" (UniqueName: \"kubernetes.io/projected/7224a54e-ce8c-4b82-9247-11376266019b-kube-api-access-nnvt7\") pod \"volume-data-source-validator-58c9f5f46d-2tt7k\" (UID: \"7224a54e-ce8c-4b82-9247-11376266019b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-2tt7k" Oct 30 18:46:05.001883 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.001868 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jh9\" (UniqueName: \"kubernetes.io/projected/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-kube-api-access-d2jh9\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:05.015578 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015557 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.015641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015580 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.015641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015595 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.015641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015610 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015667 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-multus-certs\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015669 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.015692 4789 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015705 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015732 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-conf\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015739 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.015764 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.015751 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.515735508 +0000 UTC m=+2.017645999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : secret "metrics-daemon-secret" not found Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015780 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-bin\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015785 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-var-lib-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015787 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015825 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-slash\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015865 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015891 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015912 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-conf-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015917 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015941 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015957 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-sys\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015981 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.015967 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016017 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016018 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-socket-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016019 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016047 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-kubelet\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016057 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016079 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016103 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016121 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-systemd\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016124 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016150 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-system-cni-dir\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016153 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016184 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016221 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-modprobe-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016223 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016234 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-lib-modules\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016255 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-host\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016264 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-kubelet-config\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016255 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016285 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fd86c83-2fdf-408f-a2f3-507b654c1f45-crio-socket\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016292 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016311 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016286 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-k8s-cni-cncf-io\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016317 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-run-netns\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016335 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016360 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016381 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016407 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysctl-d\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016410 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-netns\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016407 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016438 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38e48c7c-d60b-4daf-aa1a-b4578e3d910f-host-slash\") pod \"iptables-alerter-zsxjb\" (UID: \"38e48c7c-d60b-4daf-aa1a-b4578e3d910f\") " pod="openshift-network-operator/iptables-alerter-zsxjb" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016442 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016445 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016456 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016471 4789 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016500 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016469 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-multus\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016533 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.51651643 +0000 UTC m=+2.018426920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016535 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-etc-openvswitch\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016557 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.516545819 +0000 UTC m=+2.018456289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016591 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016644 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016673 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-etc-kubernetes\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016696 4789 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016714 4789 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016721 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-sysconfig\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016670 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016734 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.516722785 +0000 UTC m=+2.018633270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016761 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.516754267 +0000 UTC m=+2.018664730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016775 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016796 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016810 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-ovn\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016831 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-hostroot\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016840 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016811 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-device-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016865 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016881 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.016878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016895 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016910 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016932 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-log-socket\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016956 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-sys-fs\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016970 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.016979 4789 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016984 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.016987 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-registration-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017000 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017012 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.517001663 +0000 UTC m=+2.018912127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017008 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-cnibin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017040 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017047 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-etc-kubernetes\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017027 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-systemd-units\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017070 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d3eacfd-ea78-48c0-8968-da01c34ec561-hosts-file\") pod \"node-resolver-s65mr\" (UID: \"0d3eacfd-ea78-48c0-8968-da01c34ec561\") " pod="openshift-dns/node-resolver-s65mr" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017076 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017103 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017129 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-run\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017155 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017180 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017225 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017228 4789 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017250 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017270 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-run-systemd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017280 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017284 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6-var-lib-kubelet\") pod \"tuned-5xbl5\" (UID: \"4cf3c0f3-1cf9-4fc3-8973-ead2c2dc36a6\") " pod="openshift-cluster-node-tuning-operator/tuned-5xbl5" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017297 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.517283766 +0000 UTC m=+2.019194256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017327 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.517316094 +0000 UTC m=+2.019226562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017325 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-run-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017360 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017367 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017378 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017412 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017410 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-cni-netd\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017422 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-host-var-lib-cni-bin\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017431 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017453 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017458 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-kubelet\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017470 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017486 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-node-log\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017490 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017488 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017503 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017508 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de998501-b81d-4fa5-b100-34ee068f8b03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djnd7\" (UID: \"de998501-b81d-4fa5-b100-34ee068f8b03\") " pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017531 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017546 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017561 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017591 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fd86c83-2fdf-408f-a2f3-507b654c1f45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017604 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.517565685 +0000 UTC m=+2.019476151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017614 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f076db4-f03f-4bf3-be39-e899370f953c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fc8hw\" (UID: \"2f076db4-f03f-4bf3-be39-e899370f953c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fc8hw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017620 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-system-cni-dir\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017623 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7cee29d4-8788-4842-b8c5-a2f6e6f3996a-cnibin\") pod \"multus-additional-cni-plugins-qlzjf\" (UID: \"7cee29d4-8788-4842-b8c5-a2f6e6f3996a\") " pod="openshift-multus/multus-additional-cni-plugins-qlzjf" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017628 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c87d2cac-3415-4c41-9ba2-de3dd8ee236f-host\") pod \"node-ca-4cm6p\" (UID: \"c87d2cac-3415-4c41-9ba2-de3dd8ee236f\") " pod="openshift-image-registry/node-ca-4cm6p" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017658 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017687 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017724 4789 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.017795 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:05.517780085 +0000 UTC m=+2.019690566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017802 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9cdd95a5-32ae-4c7a-a067-bc40ad4bc826-dbus\") pod \"global-pull-secret-syncer-vnc44\" (UID: \"9cdd95a5-32ae-4c7a-a067-bc40ad4bc826\") " pod="kube-system/global-pull-secret-syncer-vnc44" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017810 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.018530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.017870 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acb8307e-d525-404e-ac04-ec75650aa4f2-multus-socket-dir-parent\") pod \"multus-pskdl\" (UID: \"acb8307e-d525-404e-ac04-ec75650aa4f2\") " pod="openshift-multus/multus-pskdl" Oct 30 18:46:05.049808 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.049787 4789 scope.go:122] "RemoveContainer" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" Oct 30 18:46:05.064040 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.063884 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" Oct 30 18:46:05.064303 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.064279 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:46:05.484709 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.484686 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s65mr_0d3eacfd-ea78-48c0-8968-da01c34ec561/dns-node-resolver/0.log" Oct 30 18:46:05.521742 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.521717 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:05.521859 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.521758 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:05.521859 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.521788 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:05.521859 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.521825 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:05.521859 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521839 4789 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:46:05.521997 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521896 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.521880335 +0000 UTC m=+3.023790819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:46:05.521997 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521903 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:05.521997 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521916 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:05.521997 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521923 4789 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:05.521997 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.521930 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.521916456 +0000 UTC m=+3.023826924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522003 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.521986495 +0000 UTC m=+3.023896976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522035 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522085 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.522072044 +0000 UTC m=+3.023982521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522098 4789 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522110 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522152 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.522137665 +0000 UTC m=+3.024048148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522178 4789 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522230 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.522199696 +0000 UTC m=+3.024110196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : secret "metrics-daemon-secret" not found Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522205 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:05.522260 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522260 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522280 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522301 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.522314 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522322 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522349 4789 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522372 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.522356174 +0000 UTC m=+3.024266677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522393 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.52238288 +0000 UTC m=+3.024293343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522402 4789 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522432 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.52242474 +0000 UTC m=+3.024335237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522432 4789 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:46:05.522620 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.522476 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:06.522463836 +0000 UTC m=+3.024374314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:46:05.719361 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.719331 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:46:05.719784 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.719769 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/1.log" Oct 30 18:46:05.719823 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.719812 4789 generic.go:356] "Generic (PLEG): container finished" podID="94f91f0e-7db9-426d-ac87-4c055d52abaa" containerID="3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0" exitCode=255 Oct 30 18:46:05.719899 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.719881 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" event={"ID":"94f91f0e-7db9-426d-ac87-4c055d52abaa","Type":"ContainerDied","Data":"3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0"} Oct 30 18:46:05.719953 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.719930 4789 scope.go:122] "RemoveContainer" containerID="6268246300a8a3a1db29e9578ecaf0662c29d873304f6015020e98719e9ffd7a" Oct 30 18:46:05.720291 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.720276 4789 scope.go:122] "RemoveContainer" containerID="3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0" Oct 30 18:46:05.720503 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:05.720480 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:05.721429 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.721409 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" event={"ID":"8db9995d-8178-4095-88a2-01ca522016d2","Type":"ContainerStarted","Data":"9bda783ca6ac75d62cd311a4da138b947f1226457f1412065c0d98d96a6e391c"} Oct 30 18:46:05.721522 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:05.721455 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8npkv" event={"ID":"8db9995d-8178-4095-88a2-01ca522016d2","Type":"ContainerStarted","Data":"ea62417ad8a443cfebe994240cc15c7c143327c8ae15b463c47fe0d6e62bc95d"} Oct 30 18:46:06.079308 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.079281 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4cm6p_c87d2cac-3415-4c41-9ba2-de3dd8ee236f/node-ca/0.log" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564520 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564577 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564642 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564679 4789 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564737 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564751 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564689 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564679 4789 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564778 4789 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564752 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs podName:8b36189e-0ae0-498e-8bd9-0391d1f3a47c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.564731874 +0000 UTC m=+5.066642342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs") pod "network-metrics-daemon-jgml5" (UID: "8b36189e-0ae0-498e-8bd9-0391d1f3a47c") : secret "metrics-daemon-secret" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564845 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.564829511 +0000 UTC m=+5.066739979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564861 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.564852603 +0000 UTC m=+5.066763073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564883 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.564875888 +0000 UTC m=+5.066786360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564917 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564949 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.564980 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.564999 4789 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.565015 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.565042 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565051 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.565036913 +0000 UTC m=+5.066947392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : secret "router-metrics-certs-default" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565110 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.565101054 +0000 UTC m=+5.067011541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.565106 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565172 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565184 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565190 4789 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565236 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls podName:f7666d1a-d160-4cc8-a277-208408706b6d nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.5652256 +0000 UTC m=+5.067136066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-lmhht" (UID: "f7666d1a-d160-4cc8-a277-208408706b6d") : secret "cluster-monitoring-operator-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565251 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.565242357 +0000 UTC m=+5.067152836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565293 4789 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565298 4789 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565327 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.565314977 +0000 UTC m=+5.067225454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:46:06.568225 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:06.565352 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:08.565335981 +0000 UTC m=+5.067246482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:06.726414 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.726393 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:46:06.878507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:06.878444 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xw9bs_dc8860e1-0ab3-4d6d-a343-1e8b4e7b3440/serve-healthcheck-canary/0.log" Oct 30 18:46:07.479673 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:07.479639 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m_1f732cf0-c45a-4146-9d07-63c3e1e15ba6/kube-storage-version-migrator-operator/0.log" Oct 30 18:46:07.730774 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:07.730702 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g22qv" event={"ID":"bdccbf5a-017a-46f7-a82e-df13cc459a30","Type":"ContainerStarted","Data":"ee47f752ce7f14681f281806de08763d9588b7f7aa344f7b7442bcc20b425e8d"} Oct 30 18:46:07.732223 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:07.732191 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nvg6w" event={"ID":"48d1db69-50e4-4cc9-a744-2ab11bd3f519","Type":"ContainerStarted","Data":"0498c1cbbded2c1f92fc7b47e27c99cba8419aadc25b01eaebd8476b28fa0c19"} Oct 30 18:46:08.584205 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584173 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:08.584389 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584225 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:08.584389 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584262 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:08.584389 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584284 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:08.584389 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584310 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:08.584389 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584341 4789 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584406 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584423 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584428 4789 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584414 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls podName:7fd86c83-2fdf-408f-a2f3-507b654c1f45 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584394289 +0000 UTC m=+9.086304754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ckq9" (UID: "7fd86c83-2fdf-408f-a2f3-507b654c1f45") : secret "insights-runtime-extractor-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584476 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584462653 +0000 UTC m=+9.086373117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584495 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584537 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584526569 +0000 UTC m=+9.086437061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584551 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584545228 +0000 UTC m=+9.086455723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584585 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584600 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584611 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:08.584636 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584644 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584629417 +0000 UTC m=+9.086539882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584663 4789 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584675 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584687 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.58468059 +0000 UTC m=+9.086591094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584727 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584756 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584822 4789 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:08.584852 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls podName:e306cc55-def6-43b0-99d2-ea6549ea2a1c nodeName:}" failed. No retries permitted until 2026-10-30 18:46:12.584842992 +0000 UTC m=+9.086753505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls") pod "dns-default-q7r4n" (UID: "e306cc55-def6-43b0-99d2-ea6549ea2a1c") : secret "dns-default-metrics-tls" not found Oct 30 18:46:08.585083 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.584862 4789 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:46:08.588804 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.588774 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b36189e-0ae0-498e-8bd9-0391d1f3a47c-metrics-certs\") pod \"network-metrics-daemon-jgml5\" (UID: \"8b36189e-0ae0-498e-8bd9-0391d1f3a47c\") " pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:08.588918 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.588832 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7666d1a-d160-4cc8-a277-208408706b6d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-lmhht\" (UID: \"f7666d1a-d160-4cc8-a277-208408706b6d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:08.588918 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.588881 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0605710c-1a4a-47f2-a7bc-4079a97e9db4-metrics-certs\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:08.649139 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.649111 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" Oct 30 18:46:08.656673 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:08.656650 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jgml5" Oct 30 18:46:09.029128 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:09.029102 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht"] Oct 30 18:46:09.069055 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:09.069029 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7666d1a_d160_4cc8_a277_208408706b6d.slice/crio-cdd4953b7f0c87bf24022dfcf0702a954697c2bc0ec987d8cd27df4d322123c1 WatchSource:0}: Error finding container cdd4953b7f0c87bf24022dfcf0702a954697c2bc0ec987d8cd27df4d322123c1: Status 404 returned error can't find the container with id cdd4953b7f0c87bf24022dfcf0702a954697c2bc0ec987d8cd27df4d322123c1 Oct 30 18:46:09.075686 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:09.075663 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jgml5"] Oct 30 18:46:09.087657 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:09.087637 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b36189e_0ae0_498e_8bd9_0391d1f3a47c.slice/crio-2341662798ba3201b239f3402a82e31ee918edf4eaa6fa37446ee234212e0106 WatchSource:0}: Error finding container 2341662798ba3201b239f3402a82e31ee918edf4eaa6fa37446ee234212e0106: Status 404 returned error can't find the container with id 2341662798ba3201b239f3402a82e31ee918edf4eaa6fa37446ee234212e0106 Oct 30 18:46:09.739596 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:09.739563 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" event={"ID":"f7666d1a-d160-4cc8-a277-208408706b6d","Type":"ContainerStarted","Data":"cdd4953b7f0c87bf24022dfcf0702a954697c2bc0ec987d8cd27df4d322123c1"} Oct 30 18:46:09.740684 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:09.740661 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgml5" event={"ID":"8b36189e-0ae0-498e-8bd9-0391d1f3a47c","Type":"ContainerStarted","Data":"2341662798ba3201b239f3402a82e31ee918edf4eaa6fa37446ee234212e0106"} Oct 30 18:46:10.573595 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:10.573560 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:46:11.749269 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:11.749237 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgml5" event={"ID":"8b36189e-0ae0-498e-8bd9-0391d1f3a47c","Type":"ContainerStarted","Data":"0b2b58b60bfc96f51d4007b62314a94067933e04b0ce8c11e817b2a53704d811"} Oct 30 18:46:11.749653 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:11.749287 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jgml5" event={"ID":"8b36189e-0ae0-498e-8bd9-0391d1f3a47c","Type":"ContainerStarted","Data":"a01b0672e1d0d924443c38e9fc2236142081095b75d9f55a5c7951def43988c7"} Oct 30 18:46:11.750722 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:11.750697 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-lmhht" event={"ID":"f7666d1a-d160-4cc8-a277-208408706b6d","Type":"ContainerStarted","Data":"e53b374f1edab30df967fa04acdf58b8b4c4beacc07ba327268483b52f73d87b"} Oct 30 18:46:12.624869 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624833 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624876 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624922 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624941 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624965 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.624981 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:12.625033 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625005 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bbc68679b-ln4qc: secret "image-registry-tls" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625063 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls podName:528cf1e4-dafb-406c-a37b-c055df14a5e1 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:20.625046354 +0000 UTC m=+17.126956859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls") pod "image-registry-bbc68679b-ln4qc" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1") : secret "image-registry-tls" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.624987 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.625121 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625068 4789 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625197 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert podName:6c6b33a9-100f-42a4-b231-8e00e3d3520b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:20.625180965 +0000 UTC m=+17.127091444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jnbnw" (UID: "6c6b33a9-100f-42a4-b231-8e00e3d3520b") : secret "networking-console-plugin-cert" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625068 4789 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625268 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle podName:0605710c-1a4a-47f2-a7bc-4079a97e9db4 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:20.625256627 +0000 UTC m=+17.127167092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle") pod "router-default-5978b965c7-kbthb" (UID: "0605710c-1a4a-47f2-a7bc-4079a97e9db4") : configmap references non-existent config key: service-ca.crt Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625134 4789 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 18:46:12.625332 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625332 4789 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f85f87669-rmcbh: secret "image-registry-tls" not found Oct 30 18:46:12.625632 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625312 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls podName:d3a136f8-4c59-458c-b2bd-c1645cb9eb28 nodeName:}" failed. No retries permitted until 2026-10-30 18:46:20.625290617 +0000 UTC m=+17.127201095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hjjwc" (UID: "d3a136f8-4c59-458c-b2bd-c1645cb9eb28") : secret "samples-operator-tls" not found Oct 30 18:46:12.625632 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:12.625410 4789 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls podName:fcb96ed0-2dfb-449a-8389-095c46386e9b nodeName:}" failed. No retries permitted until 2026-10-30 18:46:20.625395337 +0000 UTC m=+17.127305800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls") pod "image-registry-6f85f87669-rmcbh" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b") : secret "image-registry-tls" not found Oct 30 18:46:12.627203 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.627183 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fd86c83-2fdf-408f-a2f3-507b654c1f45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ckq9\" (UID: \"7fd86c83-2fdf-408f-a2f3-507b654c1f45\") " pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:12.627324 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.627309 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e306cc55-def6-43b0-99d2-ea6549ea2a1c-metrics-tls\") pod \"dns-default-q7r4n\" (UID: \"e306cc55-def6-43b0-99d2-ea6549ea2a1c\") " pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:12.726924 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.726898 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:12.861355 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.861326 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:12.862416 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:12.862397 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ckq9" Oct 30 18:46:13.123357 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.123326 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:13.123591 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.123575 4789 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:46:13.238858 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.238835 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q7r4n"] Oct 30 18:46:13.246757 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:13.246732 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode306cc55_def6_43b0_99d2_ea6549ea2a1c.slice/crio-ae0f0fd8b01b0b3a5d42ebcc92af0a523a50716678850c31d0f8b34679000e32 WatchSource:0}: Error finding container ae0f0fd8b01b0b3a5d42ebcc92af0a523a50716678850c31d0f8b34679000e32: Status 404 returned error can't find the container with id ae0f0fd8b01b0b3a5d42ebcc92af0a523a50716678850c31d0f8b34679000e32 Oct 30 18:46:13.254053 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.254031 4789 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:13.254432 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.254416 4789 scope.go:122] "RemoveContainer" containerID="3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0" Oct 30 18:46:13.254629 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:13.254605 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:13.290575 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.290548 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ckq9"] Oct 30 18:46:13.298197 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:13.298177 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd86c83_2fdf_408f_a2f3_507b654c1f45.slice/crio-9b0c85dad907a73555208ebcee5ad59b86eb01f320daaa52ed55e6dc123b3ade WatchSource:0}: Error finding container 9b0c85dad907a73555208ebcee5ad59b86eb01f320daaa52ed55e6dc123b3ade: Status 404 returned error can't find the container with id 9b0c85dad907a73555208ebcee5ad59b86eb01f320daaa52ed55e6dc123b3ade Oct 30 18:46:13.759351 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.759320 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ckq9" event={"ID":"7fd86c83-2fdf-408f-a2f3-507b654c1f45","Type":"ContainerStarted","Data":"38d458eaa54e55c4b94ae57e33c5a7f58659383e60d479179ef661dd8333a87b"} Oct 30 18:46:13.759633 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.759381 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ckq9" event={"ID":"7fd86c83-2fdf-408f-a2f3-507b654c1f45","Type":"ContainerStarted","Data":"9b0c85dad907a73555208ebcee5ad59b86eb01f320daaa52ed55e6dc123b3ade"} Oct 30 18:46:13.760451 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:13.760423 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7r4n" event={"ID":"e306cc55-def6-43b0-99d2-ea6549ea2a1c","Type":"ContainerStarted","Data":"ae0f0fd8b01b0b3a5d42ebcc92af0a523a50716678850c31d0f8b34679000e32"} Oct 30 18:46:14.205638 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:14.205606 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:14.206067 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:14.206049 4789 scope.go:122] "RemoveContainer" containerID="3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0" Oct 30 18:46:14.206289 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:14.206263 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nlzxq_openshift-console-operator(94f91f0e-7db9-426d-ac87-4c055d52abaa)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" Oct 30 18:46:14.765969 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:14.765895 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ckq9" event={"ID":"7fd86c83-2fdf-408f-a2f3-507b654c1f45","Type":"ContainerStarted","Data":"ef3e5e9abbeac62eee94d55775ff492f8ecc8a8eb0fc08db5178d1f01c831818"} Oct 30 18:46:15.667547 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.667527 4789 scope.go:122] "RemoveContainer" containerID="3e6502aaf61b8f6122a7f581296c989d134a393e2f8d52d5d1e22529411c00d0" Oct 30 18:46:15.774138 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.774035 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7r4n" event={"ID":"e306cc55-def6-43b0-99d2-ea6549ea2a1c","Type":"ContainerStarted","Data":"987164307d97f12fabcc92a51265da08973c988ba90fc149a007da1ecddb958e"} Oct 30 18:46:15.774138 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.774093 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q7r4n" event={"ID":"e306cc55-def6-43b0-99d2-ea6549ea2a1c","Type":"ContainerStarted","Data":"a1333a7b02cf171a9ac5796501831279fe9664e872dfd958f4fb26156af67406"} Oct 30 18:46:15.774138 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.774142 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:15.776597 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.776572 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:46:15.776708 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.776674 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" event={"ID":"94f91f0e-7db9-426d-ac87-4c055d52abaa","Type":"ContainerStarted","Data":"03ef5787929dd7ad7a5783209c78656b6a14353ed8af512e0a2d78ef22905cd5"} Oct 30 18:46:15.777017 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.776989 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:15.778860 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.778797 4789 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-nlzxq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body= Oct 30 18:46:15.778995 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:15.778891 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" podUID="94f91f0e-7db9-426d-ac87-4c055d52abaa" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.22:8443/readyz\": dial tcp 10.132.0.22:8443: connect: connection refused" Oct 30 18:46:16.171064 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:16.170994 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djnd7" Oct 30 18:46:16.782017 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:16.781972 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ckq9" event={"ID":"7fd86c83-2fdf-408f-a2f3-507b654c1f45","Type":"ContainerStarted","Data":"f4e32e6728f04a9a4bb388121c7a0c31b909ea44004b3f3ca40c88fb1d7c0c35"} Oct 30 18:46:16.786259 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:16.786237 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nlzxq" Oct 30 18:46:16.958730 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:16.958706 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rs2ft" Oct 30 18:46:18.194960 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:18.194922 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:46:20.697476 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.697441 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:20.697476 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.697481 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:20.697890 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.697525 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:20.697890 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.697559 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:20.697890 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.697591 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:20.698293 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.698269 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0605710c-1a4a-47f2-a7bc-4079a97e9db4-service-ca-bundle\") pod \"router-default-5978b965c7-kbthb\" (UID: \"0605710c-1a4a-47f2-a7bc-4079a97e9db4\") " pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:20.699930 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.699901 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"image-registry-6f85f87669-rmcbh\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:20.700027 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.699936 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"image-registry-bbc68679b-ln4qc\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:20.700297 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.700280 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6c6b33a9-100f-42a4-b231-8e00e3d3520b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jnbnw\" (UID: \"6c6b33a9-100f-42a4-b231-8e00e3d3520b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:20.700403 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.700382 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3a136f8-4c59-458c-b2bd-c1645cb9eb28-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hjjwc\" (UID: \"d3a136f8-4c59-458c-b2bd-c1645cb9eb28\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:20.948902 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.948834 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:20.949022 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.948923 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:20.949895 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.949880 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" Oct 30 18:46:20.955022 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.955009 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" Oct 30 18:46:20.960643 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:20.960622 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:21.393504 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.393425 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc"] Oct 30 18:46:21.446940 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.446917 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5978b965c7-kbthb"] Oct 30 18:46:21.453712 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:21.453684 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0605710c_1a4a_47f2_a7bc_4079a97e9db4.slice/crio-5e640e9e28258db83a1382f0b55ca0f8343fa94d91ecc7bad9f165bb83cb9111 WatchSource:0}: Error finding container 5e640e9e28258db83a1382f0b55ca0f8343fa94d91ecc7bad9f165bb83cb9111: Status 404 returned error can't find the container with id 5e640e9e28258db83a1382f0b55ca0f8343fa94d91ecc7bad9f165bb83cb9111 Oct 30 18:46:21.491354 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:21.491330 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb96ed0_2dfb_449a_8389_095c46386e9b.slice/crio-073149ff63cb50a683284c16f7c693d7dfcaa1705565fe2cfe97d607517013ab WatchSource:0}: Error finding container 073149ff63cb50a683284c16f7c693d7dfcaa1705565fe2cfe97d607517013ab: Status 404 returned error can't find the container with id 073149ff63cb50a683284c16f7c693d7dfcaa1705565fe2cfe97d607517013ab Oct 30 18:46:21.492358 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:21.492328 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528cf1e4_dafb_406c_a37b_c055df14a5e1.slice/crio-3922eca6685a44804f2387c7b1e17bd3f9b38e59334532578cd708b1426861a7 WatchSource:0}: Error finding container 3922eca6685a44804f2387c7b1e17bd3f9b38e59334532578cd708b1426861a7: Status 404 returned error can't find the container with id 3922eca6685a44804f2387c7b1e17bd3f9b38e59334532578cd708b1426861a7 Oct 30 18:46:21.493547 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.493475 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:46:21.494641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.494542 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:46:21.495536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.495510 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jnbnw"] Oct 30 18:46:21.507421 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:21.507386 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6b33a9_100f_42a4_b231_8e00e3d3520b.slice/crio-aaa90b6aad8caf6f1278379aeda6396f4e989ce0b1366f3cd7c6f56845b67f58 WatchSource:0}: Error finding container aaa90b6aad8caf6f1278379aeda6396f4e989ce0b1366f3cd7c6f56845b67f58: Status 404 returned error can't find the container with id aaa90b6aad8caf6f1278379aeda6396f4e989ce0b1366f3cd7c6f56845b67f58 Oct 30 18:46:21.805523 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.805489 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" event={"ID":"fcb96ed0-2dfb-449a-8389-095c46386e9b","Type":"ContainerStarted","Data":"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d"} Oct 30 18:46:21.805984 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.805543 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" event={"ID":"fcb96ed0-2dfb-449a-8389-095c46386e9b","Type":"ContainerStarted","Data":"073149ff63cb50a683284c16f7c693d7dfcaa1705565fe2cfe97d607517013ab"} Oct 30 18:46:21.805984 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.805578 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:21.807926 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.807901 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5978b965c7-kbthb" event={"ID":"0605710c-1a4a-47f2-a7bc-4079a97e9db4","Type":"ContainerStarted","Data":"261b73dff7c5bc2dd132ab28535e87178b1ff45d94ca17c3041e252eedc45b12"} Oct 30 18:46:21.808053 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.807948 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5978b965c7-kbthb" event={"ID":"0605710c-1a4a-47f2-a7bc-4079a97e9db4","Type":"ContainerStarted","Data":"5e640e9e28258db83a1382f0b55ca0f8343fa94d91ecc7bad9f165bb83cb9111"} Oct 30 18:46:21.809726 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.809703 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" event={"ID":"528cf1e4-dafb-406c-a37b-c055df14a5e1","Type":"ContainerStarted","Data":"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0"} Oct 30 18:46:21.809847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.809745 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" event={"ID":"528cf1e4-dafb-406c-a37b-c055df14a5e1","Type":"ContainerStarted","Data":"3922eca6685a44804f2387c7b1e17bd3f9b38e59334532578cd708b1426861a7"} Oct 30 18:46:21.809847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.809772 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:21.810991 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.810969 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" event={"ID":"d3a136f8-4c59-458c-b2bd-c1645cb9eb28","Type":"ContainerStarted","Data":"cbfb9c65b82953df4e92bb0f3b62116bd9e30d621f3fc34c339c7070486c26fa"} Oct 30 18:46:21.812241 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.812197 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" event={"ID":"6c6b33a9-100f-42a4-b231-8e00e3d3520b","Type":"ContainerStarted","Data":"aaa90b6aad8caf6f1278379aeda6396f4e989ce0b1366f3cd7c6f56845b67f58"} Oct 30 18:46:21.949370 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.949341 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:21.951667 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.951640 4789 patch_prober.go:29] interesting pod/router-default-5978b965c7-kbthb container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 30 18:46:21.951667 ip-10-0-136-46 kubenswrapper[4789]: [-]has-synced failed: reason withheld Oct 30 18:46:21.951667 ip-10-0-136-46 kubenswrapper[4789]: [+]process-running ok Oct 30 18:46:21.951667 ip-10-0-136-46 kubenswrapper[4789]: healthz check failed Oct 30 18:46:21.951958 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.951686 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-5978b965c7-kbthb" podUID="0605710c-1a4a-47f2-a7bc-4079a97e9db4" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 30 18:46:21.964175 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.964148 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb"] Oct 30 18:46:21.967796 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.967771 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:21.969136 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.969111 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vjgnj"] Oct 30 18:46:21.972237 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:21.972191 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.002138 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.002116 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:46:22.025530 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.025499 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 18:46:22.025673 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.025499 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bdsn9\"" Oct 30 18:46:22.027330 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.027311 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-rc2p5\"" Oct 30 18:46:22.027756 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.027741 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 18:46:22.027968 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.027954 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 18:46:22.063462 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.063414 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58f77cff9b-6fpzr"] Oct 30 18:46:22.065466 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.065450 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.082278 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.082257 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb"] Oct 30 18:46:22.083740 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.083715 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vjgnj"] Oct 30 18:46:22.109558 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.109527 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88tkx\" (UniqueName: \"kubernetes.io/projected/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-kube-api-access-88tkx\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.109664 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.109588 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f628c55a-1a6a-475e-b328-2ae83339e484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-298qb\" (UID: \"f628c55a-1a6a-475e-b328-2ae83339e484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:22.109706 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.109681 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-tmp\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.127966 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.127943 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58f77cff9b-6fpzr"] Oct 30 18:46:22.210746 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210720 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-image-registry-private-configuration\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.210868 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210758 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-tls\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.210868 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210838 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6991df4-e09d-466a-a25f-9a3a7cc638d8-ca-trust-extracted\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.210868 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210865 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-trusted-ca\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.210987 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210902 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88tkx\" (UniqueName: \"kubernetes.io/projected/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-kube-api-access-88tkx\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.210987 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210919 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-certificates\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.211090 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.210981 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-bound-sa-token\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.211090 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.211022 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcrq\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-kube-api-access-6bcrq\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.211190 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.211113 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-installation-pull-secrets\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.211190 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.211173 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f628c55a-1a6a-475e-b328-2ae83339e484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-298qb\" (UID: \"f628c55a-1a6a-475e-b328-2ae83339e484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:22.211312 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.211239 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-tmp\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.211632 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.211614 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-tmp\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.213541 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.213524 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f628c55a-1a6a-475e-b328-2ae83339e484-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-298qb\" (UID: \"f628c55a-1a6a-475e-b328-2ae83339e484\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:22.271025 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.271005 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88tkx\" (UniqueName: \"kubernetes.io/projected/7492d78b-2ed9-44bb-a4b4-8ea89fc5470f-kube-api-access-88tkx\") pod \"downloads-6dd789c8b7-vjgnj\" (UID: \"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f\") " pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.281871 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.281843 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:22.286681 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.286665 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:22.312145 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312114 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-image-registry-private-configuration\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312272 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312164 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-tls\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312272 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312189 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6991df4-e09d-466a-a25f-9a3a7cc638d8-ca-trust-extracted\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312272 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312225 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-trusted-ca\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312271 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-certificates\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312311 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-bound-sa-token\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312349 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcrq\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-kube-api-access-6bcrq\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.312397 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-installation-pull-secrets\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.313243 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.313202 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6991df4-e09d-466a-a25f-9a3a7cc638d8-ca-trust-extracted\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.313578 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.313528 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-certificates\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.313849 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.313830 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6991df4-e09d-466a-a25f-9a3a7cc638d8-trusted-ca\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.315898 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.315876 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-registry-tls\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.316065 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.316049 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-installation-pull-secrets\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.317542 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.317525 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6991df4-e09d-466a-a25f-9a3a7cc638d8-image-registry-private-configuration\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.374420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.374390 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcrq\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-kube-api-access-6bcrq\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.375019 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.374992 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6991df4-e09d-466a-a25f-9a3a7cc638d8-bound-sa-token\") pod \"image-registry-58f77cff9b-6fpzr\" (UID: \"a6991df4-e09d-466a-a25f-9a3a7cc638d8\") " pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.375154 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.375140 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:22.657633 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.657551 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb"] Oct 30 18:46:22.661380 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.661352 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vjgnj"] Oct 30 18:46:22.666171 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:22.666145 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf628c55a_1a6a_475e_b328_2ae83339e484.slice/crio-b4b83909f82b637c298da7f64fa6e9ab494103e430fd96bf07f0f957beaff92c WatchSource:0}: Error finding container b4b83909f82b637c298da7f64fa6e9ab494103e430fd96bf07f0f957beaff92c: Status 404 returned error can't find the container with id b4b83909f82b637c298da7f64fa6e9ab494103e430fd96bf07f0f957beaff92c Oct 30 18:46:22.668811 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:22.668772 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7492d78b_2ed9_44bb_a4b4_8ea89fc5470f.slice/crio-861761922c8ddab4616c788d0bb1d190a357307d4685bda60b61e467ecf9939b WatchSource:0}: Error finding container 861761922c8ddab4616c788d0bb1d190a357307d4685bda60b61e467ecf9939b: Status 404 returned error can't find the container with id 861761922c8ddab4616c788d0bb1d190a357307d4685bda60b61e467ecf9939b Oct 30 18:46:22.766830 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.766791 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58f77cff9b-6fpzr"] Oct 30 18:46:22.777064 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:22.777029 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6991df4_e09d_466a_a25f_9a3a7cc638d8.slice/crio-ab40dcee16e1f5554a3c35d75eac505153958bed44d181f98a2318eb8a5ae714 WatchSource:0}: Error finding container ab40dcee16e1f5554a3c35d75eac505153958bed44d181f98a2318eb8a5ae714: Status 404 returned error can't find the container with id ab40dcee16e1f5554a3c35d75eac505153958bed44d181f98a2318eb8a5ae714 Oct 30 18:46:22.820043 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.819965 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vjgnj" event={"ID":"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f","Type":"ContainerStarted","Data":"861761922c8ddab4616c788d0bb1d190a357307d4685bda60b61e467ecf9939b"} Oct 30 18:46:22.822044 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.821976 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" event={"ID":"f628c55a-1a6a-475e-b328-2ae83339e484","Type":"ContainerStarted","Data":"b4b83909f82b637c298da7f64fa6e9ab494103e430fd96bf07f0f957beaff92c"} Oct 30 18:46:22.824755 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.824726 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" event={"ID":"a6991df4-e09d-466a-a25f-9a3a7cc638d8","Type":"ContainerStarted","Data":"ab40dcee16e1f5554a3c35d75eac505153958bed44d181f98a2318eb8a5ae714"} Oct 30 18:46:22.951349 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:22.951320 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:23.843118 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.842350 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jnbnw" event={"ID":"6c6b33a9-100f-42a4-b231-8e00e3d3520b","Type":"ContainerStarted","Data":"6a94386dd4b30405c8452eb73a5b0ecf4d799aa2187c62cbd0adb4ad8ff9a24c"} Oct 30 18:46:23.845737 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.845704 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" event={"ID":"a6991df4-e09d-466a-a25f-9a3a7cc638d8","Type":"ContainerStarted","Data":"581775a87f82fe340882bf335a8c12eed704888d61827f9879111a66206f8415"} Oct 30 18:46:23.846017 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.845867 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:23.852864 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.852835 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" event={"ID":"d3a136f8-4c59-458c-b2bd-c1645cb9eb28","Type":"ContainerStarted","Data":"3dac212d2710dbc628f930261555bf6b1d7e74afaf4686aa9bef63fef5081071"} Oct 30 18:46:23.853006 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.852887 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hjjwc" event={"ID":"d3a136f8-4c59-458c-b2bd-c1645cb9eb28","Type":"ContainerStarted","Data":"379630464b19dbce0d641e91bb4ad199fb2077e556e75295e206f8831eb5555a"} Oct 30 18:46:23.853704 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.853685 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:23.855461 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:23.855445 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5978b965c7-kbthb" Oct 30 18:46:24.410930 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:24.410882 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" podStartSLOduration=2.410863133 podStartE2EDuration="2.410863133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:24.409841361 +0000 UTC m=+20.911751848" watchObservedRunningTime="2026-10-30 18:46:24.410863133 +0000 UTC m=+20.912773621" Oct 30 18:46:24.858939 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:24.858903 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" event={"ID":"f628c55a-1a6a-475e-b328-2ae83339e484","Type":"ContainerStarted","Data":"b11cd50f9fb4ef6c689dbf7c3d948e54412732611822ad10fc80fb7ad4c8e3a7"} Oct 30 18:46:24.859742 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:24.859690 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:24.864434 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:24.864412 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" Oct 30 18:46:24.979384 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:24.979328 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-298qb" podStartSLOduration=2.396506761 podStartE2EDuration="3.979314034s" totalImagesPullingTime="1.582807273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:24.97881506 +0000 UTC m=+21.480725550" watchObservedRunningTime="2026-10-30 18:46:24.979314034 +0000 UTC m=+21.481224520" Oct 30 18:46:25.784597 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:25.784540 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q7r4n" Oct 30 18:46:33.056874 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.056841 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t57bd"] Oct 30 18:46:33.088437 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.088404 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208503 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208466 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-tls\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208667 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208514 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-root\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208667 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208642 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-sys\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208819 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208695 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-wtmp\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208819 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208791 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-textfile\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208929 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208836 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4ck\" (UniqueName: \"kubernetes.io/projected/77e7d96a-fa2f-44c3-9d99-81e562a2292e-kube-api-access-6r4ck\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.208929 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208867 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-accelerators-collector-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.209029 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.208954 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-metrics-client-ca\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.209105 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.209085 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.250300 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.250231 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:46:33.250300 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.250253 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jrxvz\"" Oct 30 18:46:33.250300 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.250253 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:46:33.250553 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.250344 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:46:33.250553 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.250450 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:46:33.309929 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.309860 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4ck\" (UniqueName: \"kubernetes.io/projected/77e7d96a-fa2f-44c3-9d99-81e562a2292e-kube-api-access-6r4ck\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.309929 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.309912 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-accelerators-collector-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310087 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.309942 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-metrics-client-ca\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310087 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.309977 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310087 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310009 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-tls\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310203 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310080 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-root\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310203 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310160 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-sys\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310230 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-wtmp\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310314 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310274 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-root\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310314 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310280 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-textfile\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310713 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310381 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-sys\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310713 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310506 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-wtmp\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310713 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310516 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-textfile\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310713 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310638 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-accelerators-collector-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.310713 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.310657 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77e7d96a-fa2f-44c3-9d99-81e562a2292e-metrics-client-ca\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.312754 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.312734 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-tls\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.312859 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.312844 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77e7d96a-fa2f-44c3-9d99-81e562a2292e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.367964 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.367936 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4ck\" (UniqueName: \"kubernetes.io/projected/77e7d96a-fa2f-44c3-9d99-81e562a2292e-kube-api-access-6r4ck\") pod \"node-exporter-t57bd\" (UID: \"77e7d96a-fa2f-44c3-9d99-81e562a2292e\") " pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.455358 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.455328 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t57bd" Oct 30 18:46:33.496937 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:46:33.496907 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e7d96a_fa2f_44c3_9d99_81e562a2292e.slice/crio-41491abd5787a75e5f428c53d7e1db2134f2c56acb7b9e55a3361819e51e368d WatchSource:0}: Error finding container 41491abd5787a75e5f428c53d7e1db2134f2c56acb7b9e55a3361819e51e368d: Status 404 returned error can't find the container with id 41491abd5787a75e5f428c53d7e1db2134f2c56acb7b9e55a3361819e51e368d Oct 30 18:46:33.901646 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:33.901604 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t57bd" event={"ID":"77e7d96a-fa2f-44c3-9d99-81e562a2292e","Type":"ContainerStarted","Data":"41491abd5787a75e5f428c53d7e1db2134f2c56acb7b9e55a3361819e51e368d"} Oct 30 18:46:35.913229 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:35.913163 4789 generic.go:356] "Generic (PLEG): container finished" podID="77e7d96a-fa2f-44c3-9d99-81e562a2292e" containerID="37324b019055f485ff0cbe752d3a231eac707eda21ee09133c09aee34aff211d" exitCode=0 Oct 30 18:46:35.913750 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:35.913258 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t57bd" event={"ID":"77e7d96a-fa2f-44c3-9d99-81e562a2292e","Type":"ContainerDied","Data":"37324b019055f485ff0cbe752d3a231eac707eda21ee09133c09aee34aff211d"} Oct 30 18:46:36.921088 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:36.921051 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t57bd" event={"ID":"77e7d96a-fa2f-44c3-9d99-81e562a2292e","Type":"ContainerStarted","Data":"ef19c5e24567b9a9823d6a7a12435853c94ddeab56530f2e3a86efdeac002515"} Oct 30 18:46:36.921566 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:36.921116 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t57bd" event={"ID":"77e7d96a-fa2f-44c3-9d99-81e562a2292e","Type":"ContainerStarted","Data":"335f0df80ec40517b86abd3743a5ffaf1a69fbe93ed2b7d4fa35486613db1785"} Oct 30 18:46:37.082617 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:37.082569 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t57bd" podStartSLOduration=2.093310466 podStartE2EDuration="4.082553741s" totalImagesPullingTime="1.902743682s" totalInitContainerRuntime="86.499593ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:37.080372314 +0000 UTC m=+33.582282813" watchObservedRunningTime="2026-10-30 18:46:37.082553741 +0000 UTC m=+33.584464258" Oct 30 18:46:39.937968 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:39.937931 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vjgnj" event={"ID":"7492d78b-2ed9-44bb-a4b4-8ea89fc5470f","Type":"ContainerStarted","Data":"7cc67e57d9be8a3965e5765560af69d9b1a193e8c4864db16dba40d6db0cf2ec"} Oct 30 18:46:39.938415 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:39.938191 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:39.939760 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:39.939736 4789 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-vjgnj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body= Oct 30 18:46:39.939878 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:39.939787 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-vjgnj" podUID="7492d78b-2ed9-44bb-a4b4-8ea89fc5470f" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" Oct 30 18:46:40.133855 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:40.133800 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-vjgnj" podStartSLOduration=2.192507966 podStartE2EDuration="19.133786856s" totalImagesPullingTime="16.94127889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:40.094533922 +0000 UTC m=+36.596444405" watchObservedRunningTime="2026-10-30 18:46:40.133786856 +0000 UTC m=+36.635697336" Oct 30 18:46:40.577724 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:40.577688 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lbk7g" Oct 30 18:46:40.956867 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:40.956836 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vjgnj" Oct 30 18:46:42.829370 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:42.829238 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:46:43.860608 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:43.860578 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:44.865457 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:44.865423 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58f77cff9b-6fpzr" Oct 30 18:46:45.074278 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:45.074237 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:46:48.201457 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:48.201410 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:46:48.880951 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:48.880897 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" podUID="fcb96ed0-2dfb-449a-8389-095c46386e9b" containerName="registry" containerID="cri-o://00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d" gracePeriod=30 Oct 30 18:46:49.149866 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.149810 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:49.242836 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.242805 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.242863 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.242888 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.242906 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.242951 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.243037 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.243079 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.243105 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") pod \"fcb96ed0-2dfb-449a-8389-095c46386e9b\" (UID: \"fcb96ed0-2dfb-449a-8389-095c46386e9b\") " Oct 30 18:46:49.243629 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.243409 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:49.243687 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.243668 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:49.245668 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.245635 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:49.245791 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.245639 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "kube-api-access-rfbz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:49.245791 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.245717 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:49.245791 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.245749 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:49.245955 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.245817 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:49.249963 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.249931 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted" pod "fcb96ed0-2dfb-449a-8389-095c46386e9b" (UID: "fcb96ed0-2dfb-449a-8389-095c46386e9b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:49.344111 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344074 4789 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-installation-pull-secrets\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344111 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344105 4789 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-certificates\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344120 4789 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-bound-sa-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344144 4789 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fcb96ed0-2dfb-449a-8389-095c46386e9b-ca-trust-extracted\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344157 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbz2\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-kube-api-access-rfbz2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344173 4789 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fcb96ed0-2dfb-449a-8389-095c46386e9b-image-registry-private-configuration\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344186 4789 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fcb96ed0-2dfb-449a-8389-095c46386e9b-trusted-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.344346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.344199 4789 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fcb96ed0-2dfb-449a-8389-095c46386e9b-registry-tls\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:46:49.985953 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.985919 4789 generic.go:356] "Generic (PLEG): container finished" podID="fcb96ed0-2dfb-449a-8389-095c46386e9b" containerID="00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d" exitCode=0 Oct 30 18:46:49.986123 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.985965 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" Oct 30 18:46:49.986123 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.985990 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" event={"ID":"fcb96ed0-2dfb-449a-8389-095c46386e9b","Type":"ContainerDied","Data":"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d"} Oct 30 18:46:49.986123 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.986032 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f85f87669-rmcbh" event={"ID":"fcb96ed0-2dfb-449a-8389-095c46386e9b","Type":"ContainerDied","Data":"073149ff63cb50a683284c16f7c693d7dfcaa1705565fe2cfe97d607517013ab"} Oct 30 18:46:49.986123 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.986050 4789 scope.go:122] "RemoveContainer" containerID="00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d" Oct 30 18:46:49.994011 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.993994 4789 scope.go:122] "RemoveContainer" containerID="00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d" Oct 30 18:46:49.994322 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:46:49.994293 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d\": container with ID starting with 00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d not found: ID does not exist" containerID="00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d" Oct 30 18:46:49.994413 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:49.994323 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d"} err="failed to get container status \"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d\": rpc error: code = NotFound desc = could not find container \"00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d\": container with ID starting with 00cba1bb2e2bd4c9850f7ba943013dd151a4d6cf576e431c48af8c9ed05b828d not found: ID does not exist" Oct 30 18:46:50.188850 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:50.188822 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:46:50.235314 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:50.235288 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f85f87669-rmcbh"] Oct 30 18:46:51.666139 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:46:51.666105 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb96ed0-2dfb-449a-8389-095c46386e9b" path="/var/lib/kubelet/pods/fcb96ed0-2dfb-449a-8389-095c46386e9b/volumes" Oct 30 18:47:10.100314 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.100253 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" podUID="528cf1e4-dafb-406c-a37b-c055df14a5e1" containerName="registry" containerID="cri-o://ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0" gracePeriod=30 Oct 30 18:47:10.398946 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.398912 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:47:10.516697 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516663 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516697 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516701 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516945 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516722 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516945 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516759 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516945 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516792 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516945 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516843 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.516945 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516906 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.517240 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.516973 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") pod \"528cf1e4-dafb-406c-a37b-c055df14a5e1\" (UID: \"528cf1e4-dafb-406c-a37b-c055df14a5e1\") " Oct 30 18:47:10.517309 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.517276 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:10.517309 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.517298 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:10.519141 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.519109 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:10.519390 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.519360 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:10.519608 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.519574 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:10.519691 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.519584 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:10.519792 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.519768 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "kube-api-access-dthhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:10.524180 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.524157 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted" pod "528cf1e4-dafb-406c-a37b-c055df14a5e1" (UID: "528cf1e4-dafb-406c-a37b-c055df14a5e1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618525 4789 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-bound-sa-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618549 4789 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/528cf1e4-dafb-406c-a37b-c055df14a5e1-ca-trust-extracted\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618558 4789 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-tls\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618567 4789 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-trusted-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618577 4789 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-image-registry-private-configuration\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618587 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dthhg\" (UniqueName: \"kubernetes.io/projected/528cf1e4-dafb-406c-a37b-c055df14a5e1-kube-api-access-dthhg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618596 4789 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/528cf1e4-dafb-406c-a37b-c055df14a5e1-installation-pull-secrets\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:10.618602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:10.618604 4789 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/528cf1e4-dafb-406c-a37b-c055df14a5e1-registry-certificates\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:11.073107 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.073077 4789 generic.go:356] "Generic (PLEG): container finished" podID="528cf1e4-dafb-406c-a37b-c055df14a5e1" containerID="ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0" exitCode=0 Oct 30 18:47:11.073295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.073123 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" Oct 30 18:47:11.073295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.073160 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" event={"ID":"528cf1e4-dafb-406c-a37b-c055df14a5e1","Type":"ContainerDied","Data":"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0"} Oct 30 18:47:11.073295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.073232 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bbc68679b-ln4qc" event={"ID":"528cf1e4-dafb-406c-a37b-c055df14a5e1","Type":"ContainerDied","Data":"3922eca6685a44804f2387c7b1e17bd3f9b38e59334532578cd708b1426861a7"} Oct 30 18:47:11.073295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.073260 4789 scope.go:122] "RemoveContainer" containerID="ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0" Oct 30 18:47:11.082200 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.082181 4789 scope.go:122] "RemoveContainer" containerID="ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0" Oct 30 18:47:11.082455 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:47:11.082435 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0\": container with ID starting with ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0 not found: ID does not exist" containerID="ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0" Oct 30 18:47:11.082516 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.082467 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0"} err="failed to get container status \"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0\": rpc error: code = NotFound desc = could not find container \"ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0\": container with ID starting with ac85a734edab3b89fc2c7e3d0512725d1c094a16ed30984473ed572b6fabc6c0 not found: ID does not exist" Oct 30 18:47:11.272487 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.272466 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:47:11.318551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.318533 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bbc68679b-ln4qc"] Oct 30 18:47:11.665746 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:11.665718 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528cf1e4-dafb-406c-a37b-c055df14a5e1" path="/var/lib/kubelet/pods/528cf1e4-dafb-406c-a37b-c055df14a5e1/volumes" Oct 30 18:47:16.095167 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:16.095134 4789 generic.go:356] "Generic (PLEG): container finished" podID="24b20b3a-54d7-4dcd-b353-f9e28b7654c6" containerID="f2dff396677e4795d07e708fed3e1c3461ce9d35b65ad9645ad67b983cd1a5ff" exitCode=0 Oct 30 18:47:16.095641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:16.095204 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" event={"ID":"24b20b3a-54d7-4dcd-b353-f9e28b7654c6","Type":"ContainerDied","Data":"f2dff396677e4795d07e708fed3e1c3461ce9d35b65ad9645ad67b983cd1a5ff"} Oct 30 18:47:16.095641 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:16.095480 4789 scope.go:122] "RemoveContainer" containerID="f2dff396677e4795d07e708fed3e1c3461ce9d35b65ad9645ad67b983cd1a5ff" Oct 30 18:47:17.101782 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:17.101745 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gsbjx" event={"ID":"24b20b3a-54d7-4dcd-b353-f9e28b7654c6","Type":"ContainerStarted","Data":"16a56af65afdf2e002813a7cd67824ce5e573dfb7e4c34ab0cdd9ef72bc4e2bf"} Oct 30 18:47:18.198205 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:47:18.198163 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:47:20.116353 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:20.116328 4789 generic.go:356] "Generic (PLEG): container finished" podID="1f732cf0-c45a-4146-9d07-63c3e1e15ba6" containerID="71610c0ce5efbe42abfec9a7de398d32c6d635a60da28ca501b9d5d9425bd0f2" exitCode=0 Oct 30 18:47:20.116667 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:20.116401 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" event={"ID":"1f732cf0-c45a-4146-9d07-63c3e1e15ba6","Type":"ContainerDied","Data":"71610c0ce5efbe42abfec9a7de398d32c6d635a60da28ca501b9d5d9425bd0f2"} Oct 30 18:47:20.116708 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:20.116676 4789 scope.go:122] "RemoveContainer" containerID="71610c0ce5efbe42abfec9a7de398d32c6d635a60da28ca501b9d5d9425bd0f2" Oct 30 18:47:21.123062 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:21.123025 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jzn2m" event={"ID":"1f732cf0-c45a-4146-9d07-63c3e1e15ba6","Type":"ContainerStarted","Data":"a0ac0f34ec037267359fc847ef9e60e95abb26cc18e9244175299f73b9ed07e0"} Oct 30 18:47:35.185331 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:35.185294 4789 generic.go:356] "Generic (PLEG): container finished" podID="e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc" containerID="57dde4d66137ce3ce7aabce092778bf123daadc97e446708042aa7a453f237ae" exitCode=0 Oct 30 18:47:35.185693 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:35.185370 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-95sk7" event={"ID":"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc","Type":"ContainerDied","Data":"57dde4d66137ce3ce7aabce092778bf123daadc97e446708042aa7a453f237ae"} Oct 30 18:47:35.185735 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:35.185702 4789 scope.go:122] "RemoveContainer" containerID="57dde4d66137ce3ce7aabce092778bf123daadc97e446708042aa7a453f237ae" Oct 30 18:47:36.166646 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.166607 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:47:36.166834 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.166799 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" podUID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" containerName="insights-gathering" containerID="cri-o://f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c" gracePeriod=30 Oct 30 18:47:36.191123 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.191098 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-95sk7" event={"ID":"e3bd50e6-b8bb-4685-8bd7-2f1d0039fddc","Type":"ContainerStarted","Data":"2fcd87af984629a441a1566ef4622674df09890ed74945e3b9989cfd7ada7c2b"} Oct 30 18:47:36.430653 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.430627 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:47:36.603406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603377 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") pod \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " Oct 30 18:47:36.603582 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603432 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") pod \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " Oct 30 18:47:36.603582 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603464 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") pod \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " Oct 30 18:47:36.603582 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603481 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") pod \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\" (UID: \"361aa627-cb6e-4b51-96b4-f66cc3ebb23a\") " Oct 30 18:47:36.603894 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603853 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle" pod "361aa627-cb6e-4b51-96b4-f66cc3ebb23a" (UID: "361aa627-cb6e-4b51-96b4-f66cc3ebb23a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:36.603894 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603864 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path" pod "361aa627-cb6e-4b51-96b4-f66cc3ebb23a" (UID: "361aa627-cb6e-4b51-96b4-f66cc3ebb23a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:36.604057 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.603934 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle" pod "361aa627-cb6e-4b51-96b4-f66cc3ebb23a" (UID: "361aa627-cb6e-4b51-96b4-f66cc3ebb23a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:36.605838 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.605817 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882" pod "361aa627-cb6e-4b51-96b4-f66cc3ebb23a" (UID: "361aa627-cb6e-4b51-96b4-f66cc3ebb23a"). InnerVolumeSpecName "kube-api-access-zq882". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:36.704071 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.704049 4789 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-service-ca-bundle\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:36.704071 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.704071 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq882\" (UniqueName: \"kubernetes.io/projected/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-kube-api-access-zq882\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:36.704200 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.704082 4789 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-archives-path\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:36.704200 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:36.704091 4789 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/361aa627-cb6e-4b51-96b4-f66cc3ebb23a-trusted-ca-bundle\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:47:37.197094 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.197065 4789 generic.go:356] "Generic (PLEG): container finished" podID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" containerID="f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c" exitCode=2 Oct 30 18:47:37.197452 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.197118 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" Oct 30 18:47:37.197452 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.197157 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" event={"ID":"361aa627-cb6e-4b51-96b4-f66cc3ebb23a","Type":"ContainerDied","Data":"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c"} Oct 30 18:47:37.197452 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.197228 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9fgqw-rvvkn" event={"ID":"361aa627-cb6e-4b51-96b4-f66cc3ebb23a","Type":"ContainerDied","Data":"6b18f098a9e89020d70b4393f04dc410376cab9c5ed971e0413f6ad97dacc94a"} Oct 30 18:47:37.197452 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.197259 4789 scope.go:122] "RemoveContainer" containerID="f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c" Oct 30 18:47:37.206531 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.206514 4789 scope.go:122] "RemoveContainer" containerID="f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c" Oct 30 18:47:37.206806 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:47:37.206783 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c\": container with ID starting with f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c not found: ID does not exist" containerID="f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c" Oct 30 18:47:37.206880 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.206815 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c"} err="failed to get container status \"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c\": rpc error: code = NotFound desc = could not find container \"f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c\": container with ID starting with f956e7b3cec64e0a66cb17cfd3b21e982e8a59ce1054c6881973d3bd9f41ef1c not found: ID does not exist" Oct 30 18:47:37.352242 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.352192 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:47:37.397564 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.397541 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-9fgqw-rvvkn"] Oct 30 18:47:37.666549 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:47:37.666467 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" path="/var/lib/kubelet/pods/361aa627-cb6e-4b51-96b4-f66cc3ebb23a/volumes" Oct 30 18:47:48.198928 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:47:48.198885 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:48:18.197915 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:48:18.197827 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:48:48.199040 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:48:48.199001 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:49:18.197396 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:49:18.197355 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:49:48.199280 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:49:48.199176 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:50:18.198151 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:50:18.198106 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:50:48.198373 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:50:48.198311 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:51:03.674086 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:03.674055 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:51:03.674642 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:03.674600 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:51:18.197614 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:51:18.197527 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:51:48.198188 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:51:48.198131 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:51:53.641429 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641395 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp"] Oct 30 18:51:53.641889 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641871 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fcb96ed0-2dfb-449a-8389-095c46386e9b" containerName="registry" Oct 30 18:51:53.641959 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641892 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb96ed0-2dfb-449a-8389-095c46386e9b" containerName="registry" Oct 30 18:51:53.641959 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641916 4789 cpu_manager.go:414] "containerMap: removing container" podUID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" containerName="insights-gathering" Oct 30 18:51:53.641959 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641925 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" containerName="insights-gathering" Oct 30 18:51:53.641959 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641941 4789 cpu_manager.go:414] "containerMap: removing container" podUID="528cf1e4-dafb-406c-a37b-c055df14a5e1" containerName="registry" Oct 30 18:51:53.641959 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.641948 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528cf1e4-dafb-406c-a37b-c055df14a5e1" containerName="registry" Oct 30 18:51:53.642196 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.642046 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="361aa627-cb6e-4b51-96b4-f66cc3ebb23a" containerName="insights-gathering" Oct 30 18:51:53.642196 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.642059 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="528cf1e4-dafb-406c-a37b-c055df14a5e1" containerName="registry" Oct 30 18:51:53.642196 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.642076 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb96ed0-2dfb-449a-8389-095c46386e9b" containerName="registry" Oct 30 18:51:53.644064 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.644044 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.692976 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.692948 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:51:53.725071 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.725051 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-n2xmp\"" Oct 30 18:51:53.736439 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736419 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736534 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736450 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736534 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736480 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736617 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736589 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736658 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736647 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736693 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736670 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736747 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736690 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736747 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736715 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7c7j\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-kube-api-access-j7c7j\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.736833 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.736781 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.740039 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.740020 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp"] Oct 30 18:51:53.837612 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837591 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837720 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837620 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837720 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837637 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837720 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837666 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837720 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837705 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837916 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837734 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837916 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837764 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837916 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837798 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7c7j\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-kube-api-access-j7c7j\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.837916 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.837828 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.838166 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.838109 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.838166 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.838151 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.838377 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.838348 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.838441 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.838384 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.838515 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.838473 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.840298 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.840277 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.840474 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.840454 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.894934 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.894880 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.895147 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.895130 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7c7j\" (UniqueName: \"kubernetes.io/projected/45bae6ce-241c-4f8d-b724-6ea46abc05e6-kube-api-access-j7c7j\") pod \"openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp\" (UID: \"45bae6ce-241c-4f8d-b724-6ea46abc05e6\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:53.955476 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:53.955455 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:54.377345 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:54.377314 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp"] Oct 30 18:51:54.385400 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:51:54.385376 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bae6ce_241c_4f8d_b724_6ea46abc05e6.slice/crio-62af56d5af516466c8944530c0ccb3ae02c8c78ab518c40e62b534cfa90b1345 WatchSource:0}: Error finding container 62af56d5af516466c8944530c0ccb3ae02c8c78ab518c40e62b534cfa90b1345: Status 404 returned error can't find the container with id 62af56d5af516466c8944530c0ccb3ae02c8c78ab518c40e62b534cfa90b1345 Oct 30 18:51:54.387302 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:54.387284 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:51:55.223477 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.223443 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf"] Oct 30 18:51:55.226652 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.226631 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.276038 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.276007 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" event={"ID":"45bae6ce-241c-4f8d-b724-6ea46abc05e6","Type":"ContainerStarted","Data":"62af56d5af516466c8944530c0ccb3ae02c8c78ab518c40e62b534cfa90b1345"} Oct 30 18:51:55.321664 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.321639 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-g94zr\"" Oct 30 18:51:55.321767 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.321755 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:51:55.321843 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.321827 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:51:55.322883 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.322867 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf"] Oct 30 18:51:55.348285 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.348264 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ngf\" (UniqueName: \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.348388 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.348301 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.348435 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.348387 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.449507 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.449481 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ngf\" (UniqueName: \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.449602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.449534 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.449602 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.449592 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.449888 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.449871 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.449985 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.449964 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.507450 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.507396 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ngf\" (UniqueName: \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.541295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.541272 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:51:55.909426 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:55.909349 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf"] Oct 30 18:51:55.916857 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:51:55.916813 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9502ed2f_4388_4859_9c56_6a6e1cdefc29.slice/crio-fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d WatchSource:0}: Error finding container fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d: Status 404 returned error can't find the container with id fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d Oct 30 18:51:56.282625 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:56.282595 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerStarted","Data":"fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d"} Oct 30 18:51:58.188938 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.188909 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:51:58.189167 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.188977 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:51:58.189167 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.189006 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:51:58.293499 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.293464 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" event={"ID":"45bae6ce-241c-4f8d-b724-6ea46abc05e6","Type":"ContainerStarted","Data":"7075361327f46f40b67bebf0c7c1e8ac202bb215fa9e702eb9b73d9b0154b36d"} Oct 30 18:51:58.457329 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.457237 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" podStartSLOduration=1.656564044 podStartE2EDuration="5.457205281s" totalImagesPullingTime="3.800641237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:58.455939962 +0000 UTC m=+354.957850448" watchObservedRunningTime="2026-10-30 18:51:58.457205281 +0000 UTC m=+354.959115767" Oct 30 18:51:58.955556 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:58.955518 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:51:59.961840 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:51:59.961809 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:52:00.302268 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:00.302181 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:52:00.303274 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:00.303249 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp" Oct 30 18:52:03.315448 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:03.315412 4789 generic.go:356] "Generic (PLEG): container finished" podID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerID="af64a742f03bd908ade1428f3787669528347d85e64e6490e1dd211216d4842e" exitCode=0 Oct 30 18:52:03.315913 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:03.315501 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerDied","Data":"af64a742f03bd908ade1428f3787669528347d85e64e6490e1dd211216d4842e"} Oct 30 18:52:05.325794 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:05.325735 4789 generic.go:356] "Generic (PLEG): container finished" podID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerID="30ca00819b4b439a27a3d47582063ff7c8b749f8a6894ae44931407064bb956c" exitCode=0 Oct 30 18:52:05.326081 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:05.325824 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerDied","Data":"30ca00819b4b439a27a3d47582063ff7c8b749f8a6894ae44931407064bb956c"} Oct 30 18:52:12.361734 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:12.361703 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerStarted","Data":"6c7281655688cf51a542b93edb4e7a688908385fffb5105fd7e0e265c9d91cdc"} Oct 30 18:52:12.473566 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:12.473516 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" podStartSLOduration=2.51667217 podStartE2EDuration="17.473500135s" totalImagesPullingTime="14.956827965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:12.47318183 +0000 UTC m=+368.975092315" watchObservedRunningTime="2026-10-30 18:52:12.473500135 +0000 UTC m=+368.975410616" Oct 30 18:52:13.368581 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:13.368548 4789 generic.go:356] "Generic (PLEG): container finished" podID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerID="6c7281655688cf51a542b93edb4e7a688908385fffb5105fd7e0e265c9d91cdc" exitCode=0 Oct 30 18:52:13.368979 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:13.368610 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerDied","Data":"6c7281655688cf51a542b93edb4e7a688908385fffb5105fd7e0e265c9d91cdc"} Oct 30 18:52:14.496359 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.496337 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:52:14.603576 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.603551 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\") pod \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " Oct 30 18:52:14.603708 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.603599 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\") pod \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " Oct 30 18:52:14.603708 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.603650 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\" (UniqueName: \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\") pod \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\" (UID: \"9502ed2f-4388-4859-9c56-6a6e1cdefc29\") " Oct 30 18:52:14.604125 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.604104 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle" pod "9502ed2f-4388-4859-9c56-6a6e1cdefc29" (UID: "9502ed2f-4388-4859-9c56-6a6e1cdefc29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:14.605773 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.605747 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf" pod "9502ed2f-4388-4859-9c56-6a6e1cdefc29" (UID: "9502ed2f-4388-4859-9c56-6a6e1cdefc29"). InnerVolumeSpecName "kube-api-access-x5ngf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:14.608802 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.608770 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util" pod "9502ed2f-4388-4859-9c56-6a6e1cdefc29" (UID: "9502ed2f-4388-4859-9c56-6a6e1cdefc29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:14.704741 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.704718 4789 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-bundle\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:14.704741 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.704740 4789 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9502ed2f-4388-4859-9c56-6a6e1cdefc29-util\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:14.704877 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:14.704750 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5ngf\" (UniqueName: \"kubernetes.io/projected/9502ed2f-4388-4859-9c56-6a6e1cdefc29-kube-api-access-x5ngf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:15.378009 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:15.377985 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" Oct 30 18:52:15.378146 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:15.378005 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atqqdf" event={"ID":"9502ed2f-4388-4859-9c56-6a6e1cdefc29","Type":"ContainerDied","Data":"fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d"} Oct 30 18:52:15.378146 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:15.378043 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcbc2749cc61795c8a0d9d7920ba3a7a17f1cb5e2630e8753af737ea4e86411d" Oct 30 18:52:18.198072 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:52:18.198025 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:52:37.078656 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078624 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx"] Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078925 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="util" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078935 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="util" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078954 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="extract" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078959 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="extract" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078967 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="pull" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.078972 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="pull" Oct 30 18:52:37.079119 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.079035 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9502ed2f-4388-4859-9c56-6a6e1cdefc29" containerName="extract" Oct 30 18:52:37.083246 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.083225 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.205840 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.205809 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:52:37.205947 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.205882 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:52:37.205947 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.205887 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:52:37.206026 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.205938 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:52:37.206071 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.206060 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:52:37.206120 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.206109 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wxrjr\"" Oct 30 18:52:37.214129 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.214108 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx"] Oct 30 18:52:37.269432 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269409 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-manifests\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.269551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269442 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-tmp\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.269551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269469 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2mm\" (UniqueName: \"kubernetes.io/projected/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-kube-api-access-cc2mm\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.269551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269535 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.269655 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269614 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.269655 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.269640 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-metrics-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370759 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370709 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-metrics-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370839 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370760 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-manifests\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370839 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370777 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-tmp\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370906 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370883 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2mm\" (UniqueName: \"kubernetes.io/projected/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-kube-api-access-cc2mm\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370949 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370925 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.370993 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.370957 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.371120 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.371098 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-manifests\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.373144 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.373120 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-tmp\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.373377 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.373345 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-metrics-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.373462 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.373435 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.373513 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.373438 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-webhook-cert\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.430610 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.430593 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2mm\" (UniqueName: \"kubernetes.io/projected/5240cd27-fdf6-4478-8275-fcbbf05a2d8b-kube-api-access-cc2mm\") pod \"opendatahub-operator-controller-manager-575fc98dd-2ncwx\" (UID: \"5240cd27-fdf6-4478-8275-fcbbf05a2d8b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:37.696586 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:37.696560 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:38.193506 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:38.193471 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx"] Oct 30 18:52:38.200540 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:52:38.200498 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5240cd27_fdf6_4478_8275_fcbbf05a2d8b.slice/crio-9a169ad152a318ada2231480eaa55d0a90763188fe073ccb63db27636aa45e8f WatchSource:0}: Error finding container 9a169ad152a318ada2231480eaa55d0a90763188fe073ccb63db27636aa45e8f: Status 404 returned error can't find the container with id 9a169ad152a318ada2231480eaa55d0a90763188fe073ccb63db27636aa45e8f Oct 30 18:52:38.473929 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:38.473861 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" event={"ID":"5240cd27-fdf6-4478-8275-fcbbf05a2d8b","Type":"ContainerStarted","Data":"9a169ad152a318ada2231480eaa55d0a90763188fe073ccb63db27636aa45e8f"} Oct 30 18:52:42.494921 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:42.494881 4789 generic.go:356] "Generic (PLEG): container finished" podID="5240cd27-fdf6-4478-8275-fcbbf05a2d8b" containerID="d13a88f746322492d11cafbc8887d8f646fa5c07f9267813a06b7cb3c105db74" exitCode=0 Oct 30 18:52:42.495310 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:42.494969 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" event={"ID":"5240cd27-fdf6-4478-8275-fcbbf05a2d8b","Type":"ContainerDied","Data":"d13a88f746322492d11cafbc8887d8f646fa5c07f9267813a06b7cb3c105db74"} Oct 30 18:52:43.500258 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:43.500225 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" event={"ID":"5240cd27-fdf6-4478-8275-fcbbf05a2d8b","Type":"ContainerStarted","Data":"36c692dad2b2f79a9df8e2329d7590e9b1b85e5a2d04a80977e11893a9d05635"} Oct 30 18:52:43.500590 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:43.500342 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:43.655459 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:43.655383 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" podStartSLOduration=2.938201954 podStartE2EDuration="7.655372036s" totalImagesPullingTime="4.281188448s" totalInitContainerRuntime="435.981634ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:43.654423741 +0000 UTC m=+400.156334224" watchObservedRunningTime="2026-10-30 18:52:43.655372036 +0000 UTC m=+400.157282521" Oct 30 18:52:48.196290 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:52:48.196252 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:52:54.506997 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:54.506965 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-575fc98dd-2ncwx" Oct 30 18:52:57.920165 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:57.920127 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s"] Oct 30 18:52:57.925865 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:57.925838 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:52:58.022386 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.022364 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:52:58.022546 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.022412 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:52:58.022599 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.022580 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xftvn\"" Oct 30 18:52:58.023933 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.023911 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s"] Oct 30 18:52:58.030715 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.030694 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwj4w\" (UniqueName: \"kubernetes.io/projected/b4d866b2-7844-4f4a-ba79-31d75d4d8e76-kube-api-access-mwj4w\") pod \"limitador-operator-controller-manager-5778bb4449-rxf5s\" (UID: \"b4d866b2-7844-4f4a-ba79-31d75d4d8e76\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:52:58.131257 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.131233 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwj4w\" (UniqueName: \"kubernetes.io/projected/b4d866b2-7844-4f4a-ba79-31d75d4d8e76-kube-api-access-mwj4w\") pod \"limitador-operator-controller-manager-5778bb4449-rxf5s\" (UID: \"b4d866b2-7844-4f4a-ba79-31d75d4d8e76\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:52:58.197177 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.197150 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwj4w\" (UniqueName: \"kubernetes.io/projected/b4d866b2-7844-4f4a-ba79-31d75d4d8e76-kube-api-access-mwj4w\") pod \"limitador-operator-controller-manager-5778bb4449-rxf5s\" (UID: \"b4d866b2-7844-4f4a-ba79-31d75d4d8e76\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:52:58.238760 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.238735 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:52:58.609390 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:58.609322 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s"] Oct 30 18:52:58.616327 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:52:58.616301 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d866b2_7844_4f4a_ba79_31d75d4d8e76.slice/crio-cd9ce77f5a9e433ba6ff70799602d9ce4c5e0aceb26bcf515660dbdbd715268a WatchSource:0}: Error finding container cd9ce77f5a9e433ba6ff70799602d9ce4c5e0aceb26bcf515660dbdbd715268a: Status 404 returned error can't find the container with id cd9ce77f5a9e433ba6ff70799602d9ce4c5e0aceb26bcf515660dbdbd715268a Oct 30 18:52:59.570727 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:52:59.570684 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" event={"ID":"b4d866b2-7844-4f4a-ba79-31d75d4d8e76","Type":"ContainerStarted","Data":"cd9ce77f5a9e433ba6ff70799602d9ce4c5e0aceb26bcf515660dbdbd715268a"} Oct 30 18:53:02.180320 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.180283 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4"] Oct 30 18:53:02.188908 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.188888 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:02.241246 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.241222 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 18:53:02.262315 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.262290 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtlm\" (UniqueName: \"kubernetes.io/projected/36e0b317-a46f-48dc-9c68-501c0d36bef9-kube-api-access-vjtlm\") pod \"dns-operator-controller-manager-8557cbdb84-6vfk4\" (UID: \"36e0b317-a46f-48dc-9c68-501c0d36bef9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:02.287579 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.287549 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4gsnk\"" Oct 30 18:53:02.289779 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.289752 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4"] Oct 30 18:53:02.363563 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.363534 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtlm\" (UniqueName: \"kubernetes.io/projected/36e0b317-a46f-48dc-9c68-501c0d36bef9-kube-api-access-vjtlm\") pod \"dns-operator-controller-manager-8557cbdb84-6vfk4\" (UID: \"36e0b317-a46f-48dc-9c68-501c0d36bef9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:02.421492 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.421464 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtlm\" (UniqueName: \"kubernetes.io/projected/36e0b317-a46f-48dc-9c68-501c0d36bef9-kube-api-access-vjtlm\") pod \"dns-operator-controller-manager-8557cbdb84-6vfk4\" (UID: \"36e0b317-a46f-48dc-9c68-501c0d36bef9\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:02.504047 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.504015 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:02.890704 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:02.890635 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4"] Oct 30 18:53:02.924526 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:53:02.924500 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e0b317_a46f_48dc_9c68_501c0d36bef9.slice/crio-44356865e9260ec0a4e9ea96a9df591b9122cd88058d25e1d70d1527844dd72b WatchSource:0}: Error finding container 44356865e9260ec0a4e9ea96a9df591b9122cd88058d25e1d70d1527844dd72b: Status 404 returned error can't find the container with id 44356865e9260ec0a4e9ea96a9df591b9122cd88058d25e1d70d1527844dd72b Oct 30 18:53:03.589068 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:03.589032 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" event={"ID":"b4d866b2-7844-4f4a-ba79-31d75d4d8e76","Type":"ContainerStarted","Data":"54723c94cdd13197849cff9ef04e698e9c077f5cac73c4cdb0fdc1fd5823947a"} Oct 30 18:53:03.589564 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:03.589092 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:53:03.590342 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:03.590316 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" event={"ID":"36e0b317-a46f-48dc-9c68-501c0d36bef9","Type":"ContainerStarted","Data":"44356865e9260ec0a4e9ea96a9df591b9122cd88058d25e1d70d1527844dd72b"} Oct 30 18:53:03.699094 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:03.699055 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" podStartSLOduration=2.382661215 podStartE2EDuration="6.699043525s" totalImagesPullingTime="4.31638231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:03.698897182 +0000 UTC m=+420.200807682" watchObservedRunningTime="2026-10-30 18:53:03.699043525 +0000 UTC m=+420.200954009" Oct 30 18:53:06.706252 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:06.706226 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 18:53:07.496717 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.496686 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb"] Oct 30 18:53:07.500089 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.500071 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.549010 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.548985 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:53:07.549127 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.549020 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:53:07.595362 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.595342 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7b582\"" Oct 30 18:53:07.596854 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.596835 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb"] Oct 30 18:53:07.604627 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.604610 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6fl\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-kube-api-access-jl6fl\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.604698 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.604648 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.617968 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.617940 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" event={"ID":"36e0b317-a46f-48dc-9c68-501c0d36bef9","Type":"ContainerStarted","Data":"d346352baaf61fae534d031102dca59ee98487f73c520ee2683586fcc876a2a0"} Oct 30 18:53:07.618109 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.618088 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:07.705907 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.705884 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6fl\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-kube-api-access-jl6fl\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.706020 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.705936 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.731495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.731451 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" podStartSLOduration=2.001036646 podStartE2EDuration="5.731440255s" totalImagesPullingTime="3.730403609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:07.7307358 +0000 UTC m=+424.232646285" watchObservedRunningTime="2026-10-30 18:53:07.731440255 +0000 UTC m=+424.233350740" Oct 30 18:53:07.762809 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.762754 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6fl\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-kube-api-access-jl6fl\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.763030 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.763013 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2c176f1-b414-4027-9b13-a94db94a485b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-sbbvb\" (UID: \"e2c176f1-b414-4027-9b13-a94db94a485b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:07.817653 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:07.817631 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" Oct 30 18:53:08.184052 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:08.184027 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb"] Oct 30 18:53:08.191005 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:53:08.190970 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c176f1_b414_4027_9b13_a94db94a485b.slice/crio-5480f849b687fed39a4a6c9139128fd304bd33c302eef6bb5dd9e6ebba55a5ef WatchSource:0}: Error finding container 5480f849b687fed39a4a6c9139128fd304bd33c302eef6bb5dd9e6ebba55a5ef: Status 404 returned error can't find the container with id 5480f849b687fed39a4a6c9139128fd304bd33c302eef6bb5dd9e6ebba55a5ef Oct 30 18:53:08.624237 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:08.624141 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" event={"ID":"e2c176f1-b414-4027-9b13-a94db94a485b","Type":"ContainerStarted","Data":"5480f849b687fed39a4a6c9139128fd304bd33c302eef6bb5dd9e6ebba55a5ef"} Oct 30 18:53:11.830806 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.830771 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp"] Oct 30 18:53:11.834751 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.834723 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:11.884694 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.884630 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp"] Oct 30 18:53:11.928881 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.928853 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t5xns\"" Oct 30 18:53:11.948180 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.948152 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m4f\" (UniqueName: \"kubernetes.io/projected/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-kube-api-access-72m4f\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:11.948323 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:11.948239 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.049284 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.049249 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.049465 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.049350 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m4f\" (UniqueName: \"kubernetes.io/projected/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-kube-api-access-72m4f\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.049657 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.049636 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.108619 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.108551 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m4f\" (UniqueName: \"kubernetes.io/projected/05644db3-a8f4-4fad-8b6a-0713ac0c8f01-kube-api-access-72m4f\") pod \"kuadrant-operator-controller-manager-678d87766f-kkqxp\" (UID: \"05644db3-a8f4-4fad-8b6a-0713ac0c8f01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.150361 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.150336 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:12.584873 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.584838 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp"] Oct 30 18:53:12.591563 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:53:12.591531 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05644db3_a8f4_4fad_8b6a_0713ac0c8f01.slice/crio-ad29aa38c1f603d1d2871dfda305884f3592986e6581fa948f8360cc25041ea0 WatchSource:0}: Error finding container ad29aa38c1f603d1d2871dfda305884f3592986e6581fa948f8360cc25041ea0: Status 404 returned error can't find the container with id ad29aa38c1f603d1d2871dfda305884f3592986e6581fa948f8360cc25041ea0 Oct 30 18:53:12.644955 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.644930 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" event={"ID":"e2c176f1-b414-4027-9b13-a94db94a485b","Type":"ContainerStarted","Data":"91c867a24d5481908003a97c069644dbc2a45cf0022710f6d9ed7dad2af72e86"} Oct 30 18:53:12.646335 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.646316 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" event={"ID":"05644db3-a8f4-4fad-8b6a-0713ac0c8f01","Type":"ContainerStarted","Data":"ad29aa38c1f603d1d2871dfda305884f3592986e6581fa948f8360cc25041ea0"} Oct 30 18:53:12.755465 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:12.755426 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-sbbvb" podStartSLOduration=1.5277934659999999 podStartE2EDuration="5.755413878s" totalImagesPullingTime="4.227620412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:12.754902446 +0000 UTC m=+429.256812928" watchObservedRunningTime="2026-10-30 18:53:12.755413878 +0000 UTC m=+429.257324362" Oct 30 18:53:14.604571 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:14.604537 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rxf5s" Oct 30 18:53:17.675027 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:17.674989 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" event={"ID":"05644db3-a8f4-4fad-8b6a-0713ac0c8f01","Type":"ContainerStarted","Data":"92f732f05831e0998f8aab09867f9141bfdc16d9b05c4eca6b416082fd83cef8"} Oct 30 18:53:17.675420 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:17.675104 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:17.879979 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:17.879922 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" podStartSLOduration=2.371436342 podStartE2EDuration="6.879908156s" totalImagesPullingTime="4.508471814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:17.83319628 +0000 UTC m=+434.335106756" watchObservedRunningTime="2026-10-30 18:53:17.879908156 +0000 UTC m=+434.381818641" Oct 30 18:53:18.195393 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:53:18.195360 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:53:18.626346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:18.626270 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-6vfk4" Oct 30 18:53:20.087828 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.087798 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4"] Oct 30 18:53:20.091336 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.091316 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.188787 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.188759 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4"] Oct 30 18:53:20.213727 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.213704 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cae15c3-20da-4a51-8b5c-3386568b4d72-tmp\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.213844 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.213761 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48kh\" (UniqueName: \"kubernetes.io/projected/4cae15c3-20da-4a51-8b5c-3386568b4d72-kube-api-access-t48kh\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.213885 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.213832 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cae15c3-20da-4a51-8b5c-3386568b4d72-tls-certs\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.230014 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.229990 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:53:20.248366 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.248344 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vrwzm\"" Oct 30 18:53:20.248366 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.248358 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:53:20.314247 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.314225 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48kh\" (UniqueName: \"kubernetes.io/projected/4cae15c3-20da-4a51-8b5c-3386568b4d72-kube-api-access-t48kh\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.314383 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.314265 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cae15c3-20da-4a51-8b5c-3386568b4d72-tls-certs\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.314383 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.314308 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cae15c3-20da-4a51-8b5c-3386568b4d72-tmp\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.316481 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.316460 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4cae15c3-20da-4a51-8b5c-3386568b4d72-tmp\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.317320 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.317297 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cae15c3-20da-4a51-8b5c-3386568b4d72-tls-certs\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.371430 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.371379 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48kh\" (UniqueName: \"kubernetes.io/projected/4cae15c3-20da-4a51-8b5c-3386568b4d72-kube-api-access-t48kh\") pod \"kube-auth-proxy-6df7c6869c-nrlk4\" (UID: \"4cae15c3-20da-4a51-8b5c-3386568b4d72\") " pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.403558 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.403538 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" Oct 30 18:53:20.778222 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:20.778188 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4"] Oct 30 18:53:20.786326 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:53:20.786295 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cae15c3_20da_4a51_8b5c_3386568b4d72.slice/crio-6f8b8b9925acdde4b3117ea68709928464532945d9976880cf60735f461899cf WatchSource:0}: Error finding container 6f8b8b9925acdde4b3117ea68709928464532945d9976880cf60735f461899cf: Status 404 returned error can't find the container with id 6f8b8b9925acdde4b3117ea68709928464532945d9976880cf60735f461899cf Oct 30 18:53:21.693657 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:21.693618 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" event={"ID":"4cae15c3-20da-4a51-8b5c-3386568b4d72","Type":"ContainerStarted","Data":"6f8b8b9925acdde4b3117ea68709928464532945d9976880cf60735f461899cf"} Oct 30 18:53:24.710318 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:24.710282 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" event={"ID":"4cae15c3-20da-4a51-8b5c-3386568b4d72","Type":"ContainerStarted","Data":"e14bbf9b5819bea48196eefcc2196d62ec915d325b920d9f9f7a69ddbc7b95be"} Oct 30 18:53:24.821369 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:24.821320 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6df7c6869c-nrlk4" podStartSLOduration=1.68564773 podStartE2EDuration="4.821307076s" totalImagesPullingTime="3.135659346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:24.820575783 +0000 UTC m=+441.322486269" watchObservedRunningTime="2026-10-30 18:53:24.821307076 +0000 UTC m=+441.323217562" Oct 30 18:53:28.682771 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:53:28.682740 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kkqxp" Oct 30 18:53:48.195270 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:53:48.195229 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:54:18.195349 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:54:18.195260 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:54:40.651570 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.651531 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:54:40.655300 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.655277 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.699290 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.699266 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.699396 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.699303 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7kj\" (UniqueName: \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.704262 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.704247 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:54:40.750411 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.750392 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gs84s\"" Oct 30 18:54:40.750596 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.750579 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:54:40.750672 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.750661 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:54:40.752254 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.752234 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:54:40.800584 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.800559 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.800668 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.800590 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7kj\" (UniqueName: \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.803083 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.803061 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.858106 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.858078 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7kj\" (UniqueName: \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\") pod \"llmisvc-controller-manager-869d5dcc5d-sxpjx\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.882608 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.882588 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-c57ns"] Oct 30 18:54:40.886025 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.886007 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:40.901827 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.901782 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-metrics-certs\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:40.901827 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.901822 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrc4\" (UniqueName: \"kubernetes.io/projected/8a9da57a-6499-4c34-832d-a0cf66b01a3d-kube-api-access-drrc4\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:40.901936 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.901888 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-cert\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:40.935200 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.935178 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 18:54:40.935443 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.935424 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 18:54:40.935518 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.935447 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vd56j\"" Oct 30 18:54:40.967069 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.967045 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:40.982867 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:40.982840 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-c57ns"] Oct 30 18:54:41.003045 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.003022 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-metrics-certs\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.003142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.003061 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrc4\" (UniqueName: \"kubernetes.io/projected/8a9da57a-6499-4c34-832d-a0cf66b01a3d-kube-api-access-drrc4\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.003142 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.003087 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-cert\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.005539 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.005490 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-cert\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.005635 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.005574 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9da57a-6499-4c34-832d-a0cf66b01a3d-metrics-certs\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.061175 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.061150 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrc4\" (UniqueName: \"kubernetes.io/projected/8a9da57a-6499-4c34-832d-a0cf66b01a3d-kube-api-access-drrc4\") pod \"odh-model-controller-769875db56-c57ns\" (UID: \"8a9da57a-6499-4c34-832d-a0cf66b01a3d\") " pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.198188 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.198162 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:41.338592 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.338558 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:54:41.570703 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:41.570639 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-c57ns"] Oct 30 18:54:41.577173 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:54:41.577148 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9da57a_6499_4c34_832d_a0cf66b01a3d.slice/crio-b85008438f81fc8599be96d4132b159fd81d83f3e5084a31340cf79772e150a7 WatchSource:0}: Error finding container b85008438f81fc8599be96d4132b159fd81d83f3e5084a31340cf79772e150a7: Status 404 returned error can't find the container with id b85008438f81fc8599be96d4132b159fd81d83f3e5084a31340cf79772e150a7 Oct 30 18:54:42.050066 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:42.050035 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-c57ns" event={"ID":"8a9da57a-6499-4c34-832d-a0cf66b01a3d","Type":"ContainerStarted","Data":"b85008438f81fc8599be96d4132b159fd81d83f3e5084a31340cf79772e150a7"} Oct 30 18:54:42.051256 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:42.051229 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" event={"ID":"871c9c42-8c44-4b0a-8cd9-2825101b4b5c","Type":"ContainerStarted","Data":"db6a1cabba19a96a7cc32a9c93ff05d6e14dc166320d5fbe60499999d2767489"} Oct 30 18:54:45.067850 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.067809 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-c57ns" event={"ID":"8a9da57a-6499-4c34-832d-a0cf66b01a3d","Type":"ContainerStarted","Data":"bffd7f83d33619b3323eab43ffa23e8e7e09fc10167fd4ec83b28d5154aa485d"} Oct 30 18:54:45.068314 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.067916 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:54:45.069295 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.069273 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" event={"ID":"871c9c42-8c44-4b0a-8cd9-2825101b4b5c","Type":"ContainerStarted","Data":"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca"} Oct 30 18:54:45.069416 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.069329 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:54:45.179183 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.179095 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-c57ns" podStartSLOduration=2.142345151 podStartE2EDuration="5.179082273s" totalImagesPullingTime="3.036737122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:45.177632144 +0000 UTC m=+521.679542632" watchObservedRunningTime="2026-10-30 18:54:45.179082273 +0000 UTC m=+521.680992785" Oct 30 18:54:45.287431 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:45.287382 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" podStartSLOduration=1.8241968960000001 podStartE2EDuration="5.287362653s" totalImagesPullingTime="3.463165757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:45.287105013 +0000 UTC m=+521.789015500" watchObservedRunningTime="2026-10-30 18:54:45.287362653 +0000 UTC m=+521.789273138" Oct 30 18:54:48.195137 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:54:48.195102 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:54:56.078638 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:54:56.078603 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-c57ns" Oct 30 18:55:16.078350 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:16.078314 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:55:18.195927 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:55:18.195893 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:55:48.200547 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:55:48.200437 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:55:52.535025 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.534988 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-txf57"] Oct 30 18:55:52.537716 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.537689 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.576447 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.576421 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jcw\" (UniqueName: \"kubernetes.io/projected/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-kube-api-access-k4jcw\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.576557 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.576453 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-cert\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.597622 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.597597 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-txf57"] Oct 30 18:55:52.677057 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.677030 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jcw\" (UniqueName: \"kubernetes.io/projected/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-kube-api-access-k4jcw\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.677227 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.677063 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-cert\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.679448 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.679426 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-cert\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.734704 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.734680 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jcw\" (UniqueName: \"kubernetes.io/projected/0f4b3eb6-8ea7-405a-8706-b35aaa6ee132-kube-api-access-k4jcw\") pod \"llmisvc-controller-manager-6466c554d8-txf57\" (UID: \"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:52.849454 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:52.849396 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:53.222093 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:53.222062 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-txf57"] Oct 30 18:55:53.229066 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:55:53.229034 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4b3eb6_8ea7_405a_8706_b35aaa6ee132.slice/crio-6d8ea6fe1fd04368e43da19483818ec32e59dc9a21547dbb02c74fac044c0dc6 WatchSource:0}: Error finding container 6d8ea6fe1fd04368e43da19483818ec32e59dc9a21547dbb02c74fac044c0dc6: Status 404 returned error can't find the container with id 6d8ea6fe1fd04368e43da19483818ec32e59dc9a21547dbb02c74fac044c0dc6 Oct 30 18:55:53.381683 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:53.381661 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" event={"ID":"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132","Type":"ContainerStarted","Data":"6d8ea6fe1fd04368e43da19483818ec32e59dc9a21547dbb02c74fac044c0dc6"} Oct 30 18:55:54.387695 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:54.387663 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" event={"ID":"0f4b3eb6-8ea7-405a-8706-b35aaa6ee132","Type":"ContainerStarted","Data":"4d89dcd3223c361c3bc81912e11e5cf33cff5dabbbc2bf67b281531360aeb715"} Oct 30 18:55:54.388108 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:54.387770 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:55:54.498930 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:55:54.498871 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" podStartSLOduration=1.798913531 podStartE2EDuration="2.498858924s" totalImagesPullingTime="699.945393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:54.497657983 +0000 UTC m=+590.999568469" watchObservedRunningTime="2026-10-30 18:55:54.498858924 +0000 UTC m=+591.000769411" Oct 30 18:56:03.703909 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:03.703879 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:56:03.705629 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:03.705611 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 18:56:18.194921 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:56:18.194880 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:56:25.395455 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.395426 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-txf57" Oct 30 18:56:25.554016 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.553981 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:56:25.554268 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.554219 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" podUID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" containerName="manager" containerID="cri-o://0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca" gracePeriod=30 Oct 30 18:56:25.806931 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.806907 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:56:25.943409 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.943383 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\" (UniqueName: \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\") pod \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " Oct 30 18:56:25.943557 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.943421 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\" (UniqueName: \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\") pod \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\" (UID: \"871c9c42-8c44-4b0a-8cd9-2825101b4b5c\") " Oct 30 18:56:25.945601 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.945574 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj" pod "871c9c42-8c44-4b0a-8cd9-2825101b4b5c" (UID: "871c9c42-8c44-4b0a-8cd9-2825101b4b5c"). InnerVolumeSpecName "kube-api-access-2g7kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:25.945725 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:25.945698 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert" pod "871c9c42-8c44-4b0a-8cd9-2825101b4b5c" (UID: "871c9c42-8c44-4b0a-8cd9-2825101b4b5c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:26.044674 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.044640 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7kj\" (UniqueName: \"kubernetes.io/projected/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-kube-api-access-2g7kj\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:56:26.044674 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.044670 4789 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/871c9c42-8c44-4b0a-8cd9-2825101b4b5c-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 18:56:26.537905 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.537871 4789 generic.go:356] "Generic (PLEG): container finished" podID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" containerID="0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca" exitCode=0 Oct 30 18:56:26.538386 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.537914 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" Oct 30 18:56:26.538386 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.537919 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" event={"ID":"871c9c42-8c44-4b0a-8cd9-2825101b4b5c","Type":"ContainerDied","Data":"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca"} Oct 30 18:56:26.538386 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.537976 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx" event={"ID":"871c9c42-8c44-4b0a-8cd9-2825101b4b5c","Type":"ContainerDied","Data":"db6a1cabba19a96a7cc32a9c93ff05d6e14dc166320d5fbe60499999d2767489"} Oct 30 18:56:26.538386 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.538007 4789 scope.go:122] "RemoveContainer" containerID="0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca" Oct 30 18:56:26.547506 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.547488 4789 scope.go:122] "RemoveContainer" containerID="0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca" Oct 30 18:56:26.547765 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:56:26.547744 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca\": container with ID starting with 0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca not found: ID does not exist" containerID="0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca" Oct 30 18:56:26.547826 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.547773 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca"} err="failed to get container status \"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca\": rpc error: code = NotFound desc = could not find container \"0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca\": container with ID starting with 0593b5e74efc8d45aa81bf1949f144f43e589f6dd9e315179c463e345299e8ca not found: ID does not exist" Oct 30 18:56:26.699372 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.699342 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:56:26.745485 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:26.745459 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sxpjx"] Oct 30 18:56:27.666971 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:56:27.666939 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" path="/var/lib/kubelet/pods/871c9c42-8c44-4b0a-8cd9-2825101b4b5c/volumes" Oct 30 18:56:48.194054 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:56:48.194019 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:57:18.196308 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:57:18.196227 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:57:48.195902 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:57:48.195866 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:58:11.368145 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.368113 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 18:58:11.368558 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.368500 4789 cpu_manager.go:414] "containerMap: removing container" podUID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" containerName="manager" Oct 30 18:58:11.368558 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.368512 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" containerName="manager" Oct 30 18:58:11.368635 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.368573 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="871c9c42-8c44-4b0a-8cd9-2825101b4b5c" containerName="manager" Oct 30 18:58:11.371780 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.371761 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.421062 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.421033 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4d7rj\"" Oct 30 18:58:11.469496 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.469465 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 18:58:11.510041 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510010 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510041 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510043 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510262 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510072 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510262 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510234 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510284 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510346 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510330 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510457 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510355 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510457 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510401 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.510536 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.510467 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xll\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611243 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611194 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611243 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611251 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611267 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611314 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xll\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611369 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611398 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611495 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611445 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611765 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611499 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611765 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611548 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611765 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611712 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.611926 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.611893 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.612130 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.612108 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.612225 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.612125 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.612299 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.612242 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.613686 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.613657 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.613902 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.613884 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.671079 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.671009 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xll\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.672873 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.672849 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-p2prz\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:11.684406 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:11.684374 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:12.109624 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:12.109587 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 18:58:12.116616 ip-10-0-136-46 kubenswrapper[4789]: W1030 18:58:12.116592 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a47b28_6fdf_49e6_8705_4ca225fdcdbd.slice/crio-79810c3afda484c68391d8ba641e112b72b3225b09671e177a5813c0bdb2abdb WatchSource:0}: Error finding container 79810c3afda484c68391d8ba641e112b72b3225b09671e177a5813c0bdb2abdb: Status 404 returned error can't find the container with id 79810c3afda484c68391d8ba641e112b72b3225b09671e177a5813c0bdb2abdb Oct 30 18:58:12.118579 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:12.118546 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:58:12.118657 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:12.118612 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:58:12.118657 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:12.118640 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 18:58:13.023551 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.023515 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" event={"ID":"74a47b28-6fdf-49e6-8705-4ca225fdcdbd","Type":"ContainerStarted","Data":"d511ea738e3843c741cd686a74f85ac31750646e950c51a552988a576b18276e"} Oct 30 18:58:13.023972 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.023576 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" event={"ID":"74a47b28-6fdf-49e6-8705-4ca225fdcdbd","Type":"ContainerStarted","Data":"79810c3afda484c68391d8ba641e112b72b3225b09671e177a5813c0bdb2abdb"} Oct 30 18:58:13.186353 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.186300 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podStartSLOduration=2.186281356 podStartE2EDuration="2.186281356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:13.184625559 +0000 UTC m=+729.686536073" watchObservedRunningTime="2026-10-30 18:58:13.186281356 +0000 UTC m=+729.688191870" Oct 30 18:58:13.685544 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.685513 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:13.686847 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.686822 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:13.686949 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:13.686875 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:14.685429 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:14.685387 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:14.685853 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:14.685448 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:15.684782 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:15.684752 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:15.684963 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:15.684801 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:16.685110 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:16.685075 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:16.685525 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:16.685137 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:17.684920 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:17.684884 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:17.685094 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:17.684935 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:18.195053 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:58:18.195013 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:58:18.685293 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:18.685259 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-p2prz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:58:18.685474 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:18.685322 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 30 18:58:20.305555 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:20.305519 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:20.306370 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:20.306336 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:20.307099 ip-10-0-136-46 kubenswrapper[4789]: I1030 18:58:20.307081 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 18:58:48.196450 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:58:48.196367 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:59:18.195470 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:59:18.195425 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 18:59:48.195923 ip-10-0-136-46 kubenswrapper[4789]: E1030 18:59:48.195884 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:00:00.169009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.168976 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:00.172570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.172544 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.175251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.175230 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:00.178444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.178417 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.222116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.222095 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:00:00.267973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.267951 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.268145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.267977 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.268145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.268012 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4x6g\" (UniqueName: \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.268284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.268154 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24f5s\" (UniqueName: \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.268284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.268192 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:00.268284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.268242 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:00.270019 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.270006 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:00.271531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.271515 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:00.368859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.368837 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.368964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.368861 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.368964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.368898 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4x6g\" (UniqueName: \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.368964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.368922 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24f5s\" (UniqueName: \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.369509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.369492 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.369573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.369556 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.427289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.427237 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24f5s\" (UniqueName: \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\") pod \"maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.433323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.433303 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4x6g\" (UniqueName: \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\") pod \"maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.485354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.485333 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:00.492427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.492395 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:00.867286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.867256 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:00.919510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.919479 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:00.922731 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:00.922703 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a36f9c_b221_4527_bff6_376467a730b4.slice/crio-7c849f12635b7885291f866840872a1a7cec5cd0752f1ede5bb7b1158d1d31b9 WatchSource:0}: Error finding container 7c849f12635b7885291f866840872a1a7cec5cd0752f1ede5bb7b1158d1d31b9: Status 404 returned error can't find the container with id 7c849f12635b7885291f866840872a1a7cec5cd0752f1ede5bb7b1158d1d31b9 Oct 30 19:00:00.924568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:00.924552 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:00:00.965915 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:00.965894 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338a7686_9ef6_4961_8162_1c0f37d388c4.slice/crio-bad9eb5642c7b4a4a24d388e822258f46198b65bdb0bd62e44fcbff607f95ec0 WatchSource:0}: Error finding container bad9eb5642c7b4a4a24d388e822258f46198b65bdb0bd62e44fcbff607f95ec0: Status 404 returned error can't find the container with id bad9eb5642c7b4a4a24d388e822258f46198b65bdb0bd62e44fcbff607f95ec0 Oct 30 19:00:01.537444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:01.537396 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerStarted","Data":"7c849f12635b7885291f866840872a1a7cec5cd0752f1ede5bb7b1158d1d31b9"} Oct 30 19:00:01.538782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:01.538751 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" event={"ID":"338a7686-9ef6-4961-8162-1c0f37d388c4","Type":"ContainerStarted","Data":"bad9eb5642c7b4a4a24d388e822258f46198b65bdb0bd62e44fcbff607f95ec0"} Oct 30 19:00:05.560892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:05.560861 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerStarted","Data":"812d3bd1cc6473fdae328ccacd4e93b36fb1cd4e8a8ca74c2ff0cb72517e32a7"} Oct 30 19:00:05.669972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:05.669856 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" podStartSLOduration=1.516388098 podStartE2EDuration="5.66984063s" totalImagesPullingTime="4.153452532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.669694707 +0000 UTC m=+842.171605190" watchObservedRunningTime="2026-10-30 19:00:05.66984063 +0000 UTC m=+842.171751119" Oct 30 19:00:05.968045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:05.968011 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:00:05.971521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:05.971503 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.022497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.022348 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ddb8b1dd-dockercfg-l5qpk\"" Oct 30 19:00:06.022497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.022448 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ddb8b1dd\"" Oct 30 19:00:06.022741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.022722 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.022896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.022881 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2f2p\" (UniqueName: \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.028037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.028014 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:00:06.123531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.123498 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.123667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.123544 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2f2p\" (UniqueName: \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.124260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.124240 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.182016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.181985 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2f2p\" (UniqueName: \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\") pod \"payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.284521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.284444 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:06.567898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.567832 4789 generic.go:356] "Generic (PLEG): container finished" podID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerID="56ab19859320f161fb4d5bb7f014e1460a52c2c7aafdda8bfead07aa417fd82b" exitCode=7 Oct 30 19:00:06.568285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.567909 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" event={"ID":"338a7686-9ef6-4961-8162-1c0f37d388c4","Type":"ContainerDied","Data":"56ab19859320f161fb4d5bb7f014e1460a52c2c7aafdda8bfead07aa417fd82b"} Oct 30 19:00:06.568285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.568104 4789 scope.go:122] "RemoveContainer" containerID="56ab19859320f161fb4d5bb7f014e1460a52c2c7aafdda8bfead07aa417fd82b" Oct 30 19:00:06.569651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.569616 4789 generic.go:356] "Generic (PLEG): container finished" podID="23a36f9c-b221-4527-bff6-376467a730b4" containerID="812d3bd1cc6473fdae328ccacd4e93b36fb1cd4e8a8ca74c2ff0cb72517e32a7" exitCode=60 Oct 30 19:00:06.569745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.569648 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerDied","Data":"812d3bd1cc6473fdae328ccacd4e93b36fb1cd4e8a8ca74c2ff0cb72517e32a7"} Oct 30 19:00:06.569865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.569851 4789 scope.go:122] "RemoveContainer" containerID="812d3bd1cc6473fdae328ccacd4e93b36fb1cd4e8a8ca74c2ff0cb72517e32a7" Oct 30 19:00:06.658879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:06.658857 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:00:06.665452 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:06.665432 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaa9b9c_85b3_4d88_851d_1ef91c32eabb.slice/crio-347d469ec0a8455aafcbb09989ac001544b0ff3cf3cbda95c87e5f990f4f883a WatchSource:0}: Error finding container 347d469ec0a8455aafcbb09989ac001544b0ff3cf3cbda95c87e5f990f4f883a: Status 404 returned error can't find the container with id 347d469ec0a8455aafcbb09989ac001544b0ff3cf3cbda95c87e5f990f4f883a Oct 30 19:00:07.576721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.576658 4789 generic.go:356] "Generic (PLEG): container finished" podID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerID="1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f" exitCode=7 Oct 30 19:00:07.577103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.576740 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" event={"ID":"338a7686-9ef6-4961-8162-1c0f37d388c4","Type":"ContainerDied","Data":"1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f"} Oct 30 19:00:07.577103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.576807 4789 scope.go:122] "RemoveContainer" containerID="56ab19859320f161fb4d5bb7f014e1460a52c2c7aafdda8bfead07aa417fd82b" Oct 30 19:00:07.577103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.577049 4789 scope.go:122] "RemoveContainer" containerID="1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f" Oct 30 19:00:07.577335 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:07.577314 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb_odh-ai-gateway-infra(338a7686-9ef6-4961-8162-1c0f37d388c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" Oct 30 19:00:07.578803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.578783 4789 generic.go:356] "Generic (PLEG): container finished" podID="23a36f9c-b221-4527-bff6-376467a730b4" containerID="06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81" exitCode=60 Oct 30 19:00:07.578905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.578866 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerDied","Data":"06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81"} Oct 30 19:00:07.579032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.579017 4789 scope.go:122] "RemoveContainer" containerID="06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81" Oct 30 19:00:07.579203 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:07.579183 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx_odh-ai-gateway-infra(23a36f9c-b221-4527-bff6-376467a730b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" podUID="23a36f9c-b221-4527-bff6-376467a730b4" Oct 30 19:00:07.580050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.580032 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" event={"ID":"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb","Type":"ContainerStarted","Data":"347d469ec0a8455aafcbb09989ac001544b0ff3cf3cbda95c87e5f990f4f883a"} Oct 30 19:00:07.587450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:07.587433 4789 scope.go:122] "RemoveContainer" containerID="812d3bd1cc6473fdae328ccacd4e93b36fb1cd4e8a8ca74c2ff0cb72517e32a7" Oct 30 19:00:08.588966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:08.588924 4789 scope.go:122] "RemoveContainer" containerID="1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f" Oct 30 19:00:08.589438 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:08.589160 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb_odh-ai-gateway-infra(338a7686-9ef6-4961-8162-1c0f37d388c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" Oct 30 19:00:08.590553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:08.590534 4789 scope.go:122] "RemoveContainer" containerID="06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81" Oct 30 19:00:08.590738 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:08.590717 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx_odh-ai-gateway-infra(23a36f9c-b221-4527-bff6-376467a730b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" podUID="23a36f9c-b221-4527-bff6-376467a730b4" Oct 30 19:00:09.598166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:09.598131 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" event={"ID":"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb","Type":"ContainerStarted","Data":"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a"} Oct 30 19:00:09.598618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:09.598249 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:09.708128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:09.708074 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" podStartSLOduration=1.951079859 podStartE2EDuration="4.708062625s" totalImagesPullingTime="2.756982766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:09.707406983 +0000 UTC m=+846.209317469" watchObservedRunningTime="2026-10-30 19:00:09.708062625 +0000 UTC m=+846.209973111" Oct 30 19:00:10.541581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.541553 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:00:10.545275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.545254 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.594974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.594953 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df563ff7\"" Oct 30 19:00:10.595066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.594974 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df563ff7-dockercfg-cqlll\"" Oct 30 19:00:10.642812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.642786 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:00:10.665999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.665973 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.666178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.666156 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkndr\" (UniqueName: \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.767528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.767501 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkndr\" (UniqueName: \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.767632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.767536 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.768254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.768236 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.825485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.825431 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkndr\" (UniqueName: \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\") pod \"payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:10.857915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:10.857888 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:11.238944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:11.238915 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:00:11.297201 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:11.297173 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a31f850_0125_42c1_8ddb_33bade562dc2.slice/crio-4d721bde7f7a036fdada65f6435ed69d5b7bb2fbc4efcc69922e821a34e00776 WatchSource:0}: Error finding container 4d721bde7f7a036fdada65f6435ed69d5b7bb2fbc4efcc69922e821a34e00776: Status 404 returned error can't find the container with id 4d721bde7f7a036fdada65f6435ed69d5b7bb2fbc4efcc69922e821a34e00776 Oct 30 19:00:11.618379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:11.618300 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" event={"ID":"0a31f850-0125-42c1-8ddb-33bade562dc2","Type":"ContainerStarted","Data":"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde"} Oct 30 19:00:11.618379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:11.618360 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" event={"ID":"0a31f850-0125-42c1-8ddb-33bade562dc2","Type":"ContainerStarted","Data":"4d721bde7f7a036fdada65f6435ed69d5b7bb2fbc4efcc69922e821a34e00776"} Oct 30 19:00:11.618607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:11.618584 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:11.727297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:11.727235 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" podStartSLOduration=1.7272031989999999 podStartE2EDuration="1.727203199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:11.726774552 +0000 UTC m=+848.228685037" watchObservedRunningTime="2026-10-30 19:00:11.727203199 +0000 UTC m=+848.229113686" Oct 30 19:00:17.661507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:17.661441 4789 scope.go:122] "RemoveContainer" containerID="06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81" Oct 30 19:00:18.195363 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:18.195325 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:00:18.651635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:18.651608 4789 generic.go:356] "Generic (PLEG): container finished" podID="23a36f9c-b221-4527-bff6-376467a730b4" containerID="c8a01832d76f30f61323b6c0b5c3d354514307e31bd7d86e6d2512f5abad29f3" exitCode=60 Oct 30 19:00:18.651758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:18.651679 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerDied","Data":"c8a01832d76f30f61323b6c0b5c3d354514307e31bd7d86e6d2512f5abad29f3"} Oct 30 19:00:18.651758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:18.651735 4789 scope.go:122] "RemoveContainer" containerID="06e70f7c0f1a6efeada0e669fe9a4cf29586c7afb9a3712e37eb3e21fe809e81" Oct 30 19:00:18.652050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:18.652027 4789 scope.go:122] "RemoveContainer" containerID="c8a01832d76f30f61323b6c0b5c3d354514307e31bd7d86e6d2512f5abad29f3" Oct 30 19:00:18.652288 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:18.652265 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx_odh-ai-gateway-infra(23a36f9c-b221-4527-bff6-376467a730b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" podUID="23a36f9c-b221-4527-bff6-376467a730b4" Oct 30 19:00:19.669753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:19.669724 4789 scope.go:122] "RemoveContainer" containerID="1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f" Oct 30 19:00:19.773000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:19.772966 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:19.925289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:19.925235 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:20.042234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.042182 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\" (UniqueName: \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\") pod \"23a36f9c-b221-4527-bff6-376467a730b4\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " Oct 30 19:00:20.042370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.042268 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\") pod \"23a36f9c-b221-4527-bff6-376467a730b4\" (UID: \"23a36f9c-b221-4527-bff6-376467a730b4\") " Oct 30 19:00:20.042716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.042689 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca" pod "23a36f9c-b221-4527-bff6-376467a730b4" (UID: "23a36f9c-b221-4527-bff6-376467a730b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.044191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.044166 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g" pod "23a36f9c-b221-4527-bff6-376467a730b4" (UID: "23a36f9c-b221-4527-bff6-376467a730b4"). InnerVolumeSpecName "kube-api-access-g4x6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.143311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.143289 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4x6g\" (UniqueName: \"kubernetes.io/projected/23a36f9c-b221-4527-bff6-376467a730b4-kube-api-access-g4x6g\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.143402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.143314 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a36f9c-b221-4527-bff6-376467a730b4-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.608937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.608912 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:00:20.664590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.664560 4789 generic.go:356] "Generic (PLEG): container finished" podID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerID="9e2b73b35d490623bb624c4b3ff6c7bf583374573c2c86f96a3b30728dfa7fc4" exitCode=60 Oct 30 19:00:20.664738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.664630 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" event={"ID":"338a7686-9ef6-4961-8162-1c0f37d388c4","Type":"ContainerDied","Data":"9e2b73b35d490623bb624c4b3ff6c7bf583374573c2c86f96a3b30728dfa7fc4"} Oct 30 19:00:20.664738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.664683 4789 scope.go:122] "RemoveContainer" containerID="1f6dc6a642cbb80833f9f7e7f841c6b1e7b1cd35952c6fec8bd99044bc57918f" Oct 30 19:00:20.664996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.664967 4789 scope.go:122] "RemoveContainer" containerID="9e2b73b35d490623bb624c4b3ff6c7bf583374573c2c86f96a3b30728dfa7fc4" Oct 30 19:00:20.665182 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:20.665164 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb_odh-ai-gateway-infra(338a7686-9ef6-4961-8162-1c0f37d388c4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" Oct 30 19:00:20.665993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.665978 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" Oct 30 19:00:20.666091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.666026 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx" event={"ID":"23a36f9c-b221-4527-bff6-376467a730b4","Type":"ContainerDied","Data":"7c849f12635b7885291f866840872a1a7cec5cd0752f1ede5bb7b1158d1d31b9"} Oct 30 19:00:20.676155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.676002 4789 scope.go:122] "RemoveContainer" containerID="c8a01832d76f30f61323b6c0b5c3d354514307e31bd7d86e6d2512f5abad29f3" Oct 30 19:00:20.931927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.931852 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:20.934733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:20.934709 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e7b96f88-29889780-n4fjx"] Oct 30 19:00:21.666705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:21.666671 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a36f9c-b221-4527-bff6-376467a730b4" path="/var/lib/kubelet/pods/23a36f9c-b221-4527-bff6-376467a730b4/volumes" Oct 30 19:00:21.837930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:21.837900 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:21.971830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:21.971802 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:22.059447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.059421 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\") pod \"338a7686-9ef6-4961-8162-1c0f37d388c4\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " Oct 30 19:00:22.059578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.059561 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\" (UniqueName: \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\") pod \"338a7686-9ef6-4961-8162-1c0f37d388c4\" (UID: \"338a7686-9ef6-4961-8162-1c0f37d388c4\") " Oct 30 19:00:22.059828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.059799 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca" pod "338a7686-9ef6-4961-8162-1c0f37d388c4" (UID: "338a7686-9ef6-4961-8162-1c0f37d388c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.061547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.061526 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s" pod "338a7686-9ef6-4961-8162-1c0f37d388c4" (UID: "338a7686-9ef6-4961-8162-1c0f37d388c4"). InnerVolumeSpecName "kube-api-access-24f5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.160817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.160794 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24f5s\" (UniqueName: \"kubernetes.io/projected/338a7686-9ef6-4961-8162-1c0f37d388c4-kube-api-access-24f5s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:22.160922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.160820 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338a7686-9ef6-4961-8162-1c0f37d388c4-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:22.627765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.627742 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:00:22.677985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.677957 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" event={"ID":"338a7686-9ef6-4961-8162-1c0f37d388c4","Type":"ContainerDied","Data":"bad9eb5642c7b4a4a24d388e822258f46198b65bdb0bd62e44fcbff607f95ec0"} Oct 30 19:00:22.677985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.677985 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb" Oct 30 19:00:22.678190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.678004 4789 scope.go:122] "RemoveContainer" containerID="9e2b73b35d490623bb624c4b3ff6c7bf583374573c2c86f96a3b30728dfa7fc4" Oct 30 19:00:22.842011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.841983 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:22.844121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:22.844099 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-45e6849e-29889780-5m9cb"] Oct 30 19:00:23.667822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:23.667788 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" path="/var/lib/kubelet/pods/338a7686-9ef6-4961-8162-1c0f37d388c4/volumes" Oct 30 19:00:39.001251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001204 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8"] Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001609 4789 cpu_manager.go:414] "containerMap: removing container" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001620 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001629 4789 cpu_manager.go:414] "containerMap: removing container" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001634 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001643 4789 cpu_manager.go:414] "containerMap: removing container" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001648 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001657 4789 cpu_manager.go:414] "containerMap: removing container" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001662 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001672 4789 cpu_manager.go:414] "containerMap: removing container" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001677 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001734 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001743 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001749 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.001929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.001759 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.006706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.006684 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.009629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.009606 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.009736 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.009652 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.009807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.009780 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df8gh\" (UniqueName: \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.057892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.057858 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:39.104427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.104391 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:39.104597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.104434 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:39.104597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.104434 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:00:39.106150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.106129 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8"] Oct 30 19:00:39.110343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.110319 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.110480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.110355 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.110480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.110403 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df8gh\" (UniqueName: \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.111141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.111116 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.112907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.112877 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.170070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.170041 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df8gh\" (UniqueName: \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\") pod \"maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.317986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.317909 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:39.368997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.368964 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s"] Oct 30 19:00:39.369642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.369620 4789 cpu_manager.go:414] "containerMap: removing container" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.369708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.369645 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.369808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.369796 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="338a7686-9ef6-4961-8162-1c0f37d388c4" containerName="cleanup" Oct 30 19:00:39.369846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.369816 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a36f9c-b221-4527-bff6-376467a730b4" containerName="cleanup" Oct 30 19:00:39.372346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.372325 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.412781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.412751 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.412925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.412811 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcldn\" (UniqueName: \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.412991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.412935 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.426528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.426503 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s"] Oct 30 19:00:39.513286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.513255 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.513457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.513319 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcldn\" (UniqueName: \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.513457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.513369 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.513925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.513901 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.515607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.515579 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.571014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.570952 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcldn\" (UniqueName: \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\") pod \"maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.690470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.690442 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:39.698455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.698424 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8"] Oct 30 19:00:39.736719 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:39.736688 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b9f609_a7fa_4487_b8e4_11b7105a398b.slice/crio-d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9 WatchSource:0}: Error finding container d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9: Status 404 returned error can't find the container with id d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9 Oct 30 19:00:39.764745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:39.764715 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" event={"ID":"03b9f609-a7fa-4487-b8e4-11b7105a398b","Type":"ContainerStarted","Data":"d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9"} Oct 30 19:00:40.065471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.065439 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s"] Oct 30 19:00:40.071743 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:40.071716 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171f7205_60d7_41e9_b344_0453cbc3567d.slice/crio-5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f WatchSource:0}: Error finding container 5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f: Status 404 returned error can't find the container with id 5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f Oct 30 19:00:40.771132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.771092 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" event={"ID":"171f7205-60d7-41e9-b344-0453cbc3567d","Type":"ContainerStarted","Data":"d7f02e8a7492e88ffa8d61ebbed1b0ee5a36887fe555a05d2dd03c619acad60d"} Oct 30 19:00:40.771316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.771152 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" event={"ID":"171f7205-60d7-41e9-b344-0453cbc3567d","Type":"ContainerStarted","Data":"5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f"} Oct 30 19:00:40.772693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.772662 4789 generic.go:356] "Generic (PLEG): container finished" podID="03b9f609-a7fa-4487-b8e4-11b7105a398b" containerID="de3d19b143b9eb6afbace88413c32721a1a027227ed01d9a5dc836bb5d495687" exitCode=0 Oct 30 19:00:40.772783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.772732 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" event={"ID":"03b9f609-a7fa-4487-b8e4-11b7105a398b","Type":"ContainerDied","Data":"de3d19b143b9eb6afbace88413c32721a1a027227ed01d9a5dc836bb5d495687"} Oct 30 19:00:40.881801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:40.881744 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" podStartSLOduration=1.8817285240000001 podStartE2EDuration="1.881728524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:40.880669138 +0000 UTC m=+877.382579623" watchObservedRunningTime="2026-10-30 19:00:40.881728524 +0000 UTC m=+877.383639004" Oct 30 19:00:41.778179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.778144 4789 generic.go:356] "Generic (PLEG): container finished" podID="171f7205-60d7-41e9-b344-0453cbc3567d" containerID="d7f02e8a7492e88ffa8d61ebbed1b0ee5a36887fe555a05d2dd03c619acad60d" exitCode=0 Oct 30 19:00:41.778633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.778239 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" event={"ID":"171f7205-60d7-41e9-b344-0453cbc3567d","Type":"ContainerDied","Data":"d7f02e8a7492e88ffa8d61ebbed1b0ee5a36887fe555a05d2dd03c619acad60d"} Oct 30 19:00:41.923848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.923819 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:41.931436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.931414 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\") pod \"03b9f609-a7fa-4487-b8e4-11b7105a398b\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " Oct 30 19:00:41.931568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.931455 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\" (UniqueName: \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\") pod \"03b9f609-a7fa-4487-b8e4-11b7105a398b\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " Oct 30 19:00:41.931568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.931485 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\") pod \"03b9f609-a7fa-4487-b8e4-11b7105a398b\" (UID: \"03b9f609-a7fa-4487-b8e4-11b7105a398b\") " Oct 30 19:00:41.931739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.931718 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca" pod "03b9f609-a7fa-4487-b8e4-11b7105a398b" (UID: "03b9f609-a7fa-4487-b8e4-11b7105a398b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:41.933465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.933438 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh" pod "03b9f609-a7fa-4487-b8e4-11b7105a398b" (UID: "03b9f609-a7fa-4487-b8e4-11b7105a398b"). InnerVolumeSpecName "kube-api-access-df8gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:41.933562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:41.933462 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca" pod "03b9f609-a7fa-4487-b8e4-11b7105a398b" (UID: "03b9f609-a7fa-4487-b8e4-11b7105a398b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:42.032395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.032325 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:42.032395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.032350 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df8gh\" (UniqueName: \"kubernetes.io/projected/03b9f609-a7fa-4487-b8e4-11b7105a398b-kube-api-access-df8gh\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:42.032395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.032360 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b9f609-a7fa-4487-b8e4-11b7105a398b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:42.783957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.783918 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" Oct 30 19:00:42.783957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.783950 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8" event={"ID":"03b9f609-a7fa-4487-b8e4-11b7105a398b","Type":"ContainerDied","Data":"d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9"} Oct 30 19:00:42.784399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.784005 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70e53939a22cef41b0e977aac7fe9af850b45601a8079d246edeb4db439a0a9" Oct 30 19:00:42.922000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.921976 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:42.939988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.939964 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\") pod \"171f7205-60d7-41e9-b344-0453cbc3567d\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " Oct 30 19:00:42.940047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.940007 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\" (UniqueName: \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\") pod \"171f7205-60d7-41e9-b344-0453cbc3567d\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " Oct 30 19:00:42.940047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.940034 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\") pod \"171f7205-60d7-41e9-b344-0453cbc3567d\" (UID: \"171f7205-60d7-41e9-b344-0453cbc3567d\") " Oct 30 19:00:42.940409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.940386 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca" pod "171f7205-60d7-41e9-b344-0453cbc3567d" (UID: "171f7205-60d7-41e9-b344-0453cbc3567d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:42.942079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.942056 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn" pod "171f7205-60d7-41e9-b344-0453cbc3567d" (UID: "171f7205-60d7-41e9-b344-0453cbc3567d"). InnerVolumeSpecName "kube-api-access-qcldn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:42.942239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:42.942202 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca" pod "171f7205-60d7-41e9-b344-0453cbc3567d" (UID: "171f7205-60d7-41e9-b344-0453cbc3567d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:43.040921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.040845 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:43.040921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.040882 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcldn\" (UniqueName: \"kubernetes.io/projected/171f7205-60d7-41e9-b344-0453cbc3567d-kube-api-access-qcldn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:43.040921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.040899 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/171f7205-60d7-41e9-b344-0453cbc3567d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:43.789634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.789602 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" event={"ID":"171f7205-60d7-41e9-b344-0453cbc3567d","Type":"ContainerDied","Data":"5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f"} Oct 30 19:00:43.790055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.789641 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s" Oct 30 19:00:43.790055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:43.789651 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc1afdb815b26036299efb251eaa3ac4e761ea87c47ff0863542a2b26b4e22f" Oct 30 19:00:44.822733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.822700 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8"] Oct 30 19:00:44.823115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823078 4789 cpu_manager.go:414] "containerMap: removing container" podUID="171f7205-60d7-41e9-b344-0453cbc3567d" containerName="revoke-keys" Oct 30 19:00:44.823115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823089 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171f7205-60d7-41e9-b344-0453cbc3567d" containerName="revoke-keys" Oct 30 19:00:44.823115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823114 4789 cpu_manager.go:414] "containerMap: removing container" podUID="03b9f609-a7fa-4487-b8e4-11b7105a398b" containerName="revoke-keys" Oct 30 19:00:44.823240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823119 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b9f609-a7fa-4487-b8e4-11b7105a398b" containerName="revoke-keys" Oct 30 19:00:44.823240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823177 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b9f609-a7fa-4487-b8e4-11b7105a398b" containerName="revoke-keys" Oct 30 19:00:44.823240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.823190 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="171f7205-60d7-41e9-b344-0453cbc3567d" containerName="revoke-keys" Oct 30 19:00:44.826611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.826594 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.854724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.854694 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.854817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.854728 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257nq\" (UniqueName: \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.854817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.854805 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.875739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.875712 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:44.875881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.875867 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:44.922087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.922057 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:44.922263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.922100 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:00:44.924010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.923989 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8"] Oct 30 19:00:44.955430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.955401 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.955430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.955433 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257nq\" (UniqueName: \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.955649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.955483 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.956051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.956028 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:44.957731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:44.957709 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:45.013760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:45.013735 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257nq\" (UniqueName: \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\") pod \"maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:45.139496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:45.139407 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:45.511152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:45.511118 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8"] Oct 30 19:00:45.517655 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:45.517624 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05c0143_c5cc_4cb2_a03c_eafc08d08afe.slice/crio-ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1 WatchSource:0}: Error finding container ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1: Status 404 returned error can't find the container with id ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1 Oct 30 19:00:45.800089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:45.800060 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" event={"ID":"f05c0143-c5cc-4cb2-a03c-eafc08d08afe","Type":"ContainerStarted","Data":"ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1"} Oct 30 19:00:46.806180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:46.806146 4789 generic.go:356] "Generic (PLEG): container finished" podID="f05c0143-c5cc-4cb2-a03c-eafc08d08afe" containerID="9927e04d0e737f6ba3959431607e97b410ff3ed3f0f8248557ce3de794ea429d" exitCode=0 Oct 30 19:00:46.806597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:46.806241 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" event={"ID":"f05c0143-c5cc-4cb2-a03c-eafc08d08afe","Type":"ContainerDied","Data":"9927e04d0e737f6ba3959431607e97b410ff3ed3f0f8248557ce3de794ea429d"} Oct 30 19:00:47.946414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:47.946390 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:47.977142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:47.977109 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\" (UniqueName: \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\") pod \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " Oct 30 19:00:47.979136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:47.979112 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq" pod "f05c0143-c5cc-4cb2-a03c-eafc08d08afe" (UID: "f05c0143-c5cc-4cb2-a03c-eafc08d08afe"). InnerVolumeSpecName "kube-api-access-257nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:48.078427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.078345 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\") pod \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " Oct 30 19:00:48.078427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.078396 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\") pod \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\" (UID: \"f05c0143-c5cc-4cb2-a03c-eafc08d08afe\") " Oct 30 19:00:48.078710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.078690 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257nq\" (UniqueName: \"kubernetes.io/projected/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-kube-api-access-257nq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:48.078805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.078781 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca" pod "f05c0143-c5cc-4cb2-a03c-eafc08d08afe" (UID: "f05c0143-c5cc-4cb2-a03c-eafc08d08afe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:48.080495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.080466 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca" pod "f05c0143-c5cc-4cb2-a03c-eafc08d08afe" (UID: "f05c0143-c5cc-4cb2-a03c-eafc08d08afe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:48.179677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.179647 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:48.179677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.179676 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f05c0143-c5cc-4cb2-a03c-eafc08d08afe-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:48.195868 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:00:48.195825 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:00:48.817656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.817629 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" Oct 30 19:00:48.819740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.817655 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8" event={"ID":"f05c0143-c5cc-4cb2-a03c-eafc08d08afe","Type":"ContainerDied","Data":"ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1"} Oct 30 19:00:48.819740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:48.817701 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff70bee235b54fc7ae6fdc42924c16184b06e40512c4d2f9371a28c0f8e0dad1" Oct 30 19:00:50.532134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.532099 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml"] Oct 30 19:00:50.532634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.532493 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f05c0143-c5cc-4cb2-a03c-eafc08d08afe" containerName="revoke-keys" Oct 30 19:00:50.532634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.532504 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05c0143-c5cc-4cb2-a03c-eafc08d08afe" containerName="revoke-keys" Oct 30 19:00:50.532634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.532611 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05c0143-c5cc-4cb2-a03c-eafc08d08afe" containerName="revoke-keys" Oct 30 19:00:50.535706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.535688 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.595330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.595305 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.595330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.595337 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsfq\" (UniqueName: \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.595549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.595477 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.630897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.630871 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:00:50.630897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.630888 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:00:50.630897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.630871 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:00:50.631092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.630930 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:00:50.632809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.632791 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml"] Oct 30 19:00:50.695828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.695805 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.695939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.695846 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.695939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.695863 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsfq\" (UniqueName: \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.696433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.696416 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.698037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.698020 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.754051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.754022 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsfq\" (UniqueName: \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\") pod \"maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:50.847885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:50.847833 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:51.218811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:51.218779 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml"] Oct 30 19:00:51.280167 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:00:51.280129 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3cb466_8629_4b1e_81c9_518264443bc3.slice/crio-b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d WatchSource:0}: Error finding container b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d: Status 404 returned error can't find the container with id b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d Oct 30 19:00:51.833963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:51.833885 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" event={"ID":"8c3cb466-8629-4b1e-81c9-518264443bc3","Type":"ContainerStarted","Data":"4abd74d2b8645b0beef7901e13057d7697f17898f2adc9066805d86c5cf6382a"} Oct 30 19:00:51.833963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:51.833939 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" event={"ID":"8c3cb466-8629-4b1e-81c9-518264443bc3","Type":"ContainerStarted","Data":"b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d"} Oct 30 19:00:51.943940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:51.943893 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" podStartSLOduration=1.943879828 podStartE2EDuration="1.943879828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:51.942837964 +0000 UTC m=+888.444748460" watchObservedRunningTime="2026-10-30 19:00:51.943879828 +0000 UTC m=+888.445790313" Oct 30 19:00:52.842261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:52.842224 4789 generic.go:356] "Generic (PLEG): container finished" podID="8c3cb466-8629-4b1e-81c9-518264443bc3" containerID="4abd74d2b8645b0beef7901e13057d7697f17898f2adc9066805d86c5cf6382a" exitCode=0 Oct 30 19:00:52.842677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:52.842295 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" event={"ID":"8c3cb466-8629-4b1e-81c9-518264443bc3","Type":"ContainerDied","Data":"4abd74d2b8645b0beef7901e13057d7697f17898f2adc9066805d86c5cf6382a"} Oct 30 19:00:53.991311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:53.991285 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:54.021249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.021204 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\") pod \"8c3cb466-8629-4b1e-81c9-518264443bc3\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " Oct 30 19:00:54.021358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.021288 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\") pod \"8c3cb466-8629-4b1e-81c9-518264443bc3\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " Oct 30 19:00:54.021681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.021652 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca" pod "8c3cb466-8629-4b1e-81c9-518264443bc3" (UID: "8c3cb466-8629-4b1e-81c9-518264443bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:54.023170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.023149 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca" pod "8c3cb466-8629-4b1e-81c9-518264443bc3" (UID: "8c3cb466-8629-4b1e-81c9-518264443bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:54.122591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.122529 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\" (UniqueName: \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\") pod \"8c3cb466-8629-4b1e-81c9-518264443bc3\" (UID: \"8c3cb466-8629-4b1e-81c9-518264443bc3\") " Oct 30 19:00:54.122704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.122690 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:54.122747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.122703 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3cb466-8629-4b1e-81c9-518264443bc3-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:54.124473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.124452 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq" pod "8c3cb466-8629-4b1e-81c9-518264443bc3" (UID: "8c3cb466-8629-4b1e-81c9-518264443bc3"). InnerVolumeSpecName "kube-api-access-mzsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:54.223289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.223259 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsfq\" (UniqueName: \"kubernetes.io/projected/8c3cb466-8629-4b1e-81c9-518264443bc3-kube-api-access-mzsfq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:54.852517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.852487 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" Oct 30 19:00:54.852688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.852486 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml" event={"ID":"8c3cb466-8629-4b1e-81c9-518264443bc3","Type":"ContainerDied","Data":"b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d"} Oct 30 19:00:54.852688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:54.852614 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68b5a4de4c97db0339f89670faa88648956c775ec9ff231a5125be0d382f13d" Oct 30 19:00:59.691489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:59.691444 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8"] Oct 30 19:00:59.737857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:00:59.737823 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-45e6849e-trjb8"] Oct 30 19:01:00.813886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:00.813848 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8"] Oct 30 19:01:00.859867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:00.859834 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4e8ca3e-n5ws8"] Oct 30 19:01:01.192000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.191970 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh"] Oct 30 19:01:01.192401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.192388 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8c3cb466-8629-4b1e-81c9-518264443bc3" containerName="revoke-keys" Oct 30 19:01:01.192456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.192402 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3cb466-8629-4b1e-81c9-518264443bc3" containerName="revoke-keys" Oct 30 19:01:01.192492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.192461 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3cb466-8629-4b1e-81c9-518264443bc3" containerName="revoke-keys" Oct 30 19:01:01.195652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.195630 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.286198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.286164 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.286381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.286347 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.286381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.286377 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhw6m\" (UniqueName: \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.290882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.290854 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:01:01.290882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.290871 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:01:01.290882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.290873 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:01:01.291051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.290896 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:01:01.292789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.292770 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh"] Oct 30 19:01:01.387348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.387316 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.387537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.387423 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.387537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.387445 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhw6m\" (UniqueName: \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.387991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.387970 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.389830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.389807 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.445837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.445757 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhw6m\" (UniqueName: \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\") pod \"maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.509313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.509285 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:01.667281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.667238 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b9f609-a7fa-4487-b8e4-11b7105a398b" path="/var/lib/kubelet/pods/03b9f609-a7fa-4487-b8e4-11b7105a398b/volumes" Oct 30 19:01:01.667755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.667733 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05c0143-c5cc-4cb2-a03c-eafc08d08afe" path="/var/lib/kubelet/pods/f05c0143-c5cc-4cb2-a03c-eafc08d08afe/volumes" Oct 30 19:01:01.884591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.884489 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh"] Oct 30 19:01:01.890788 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:01:01.890757 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79c8385_25d9_4a8e_a552_fce14c2402e7.slice/crio-f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6 WatchSource:0}: Error finding container f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6: Status 404 returned error can't find the container with id f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6 Oct 30 19:01:01.898877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:01.898855 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" event={"ID":"e79c8385-25d9-4a8e-a552-fce14c2402e7","Type":"ContainerStarted","Data":"f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6"} Oct 30 19:01:02.905642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:02.905605 4789 generic.go:356] "Generic (PLEG): container finished" podID="e79c8385-25d9-4a8e-a552-fce14c2402e7" containerID="109dec1fad3e3524313d10c25fb9361708273803e6dec87ee34fa9c620d0ee5d" exitCode=0 Oct 30 19:01:02.906029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:02.905679 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" event={"ID":"e79c8385-25d9-4a8e-a552-fce14c2402e7","Type":"ContainerDied","Data":"109dec1fad3e3524313d10c25fb9361708273803e6dec87ee34fa9c620d0ee5d"} Oct 30 19:01:03.737546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:03.737514 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:01:03.748498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:03.748462 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:01:03.865118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:03.865088 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:01:04.050606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.050578 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:04.110832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.110806 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\") pod \"e79c8385-25d9-4a8e-a552-fce14c2402e7\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " Oct 30 19:01:04.110990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.110843 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\" (UniqueName: \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\") pod \"e79c8385-25d9-4a8e-a552-fce14c2402e7\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " Oct 30 19:01:04.110990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.110897 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\") pod \"e79c8385-25d9-4a8e-a552-fce14c2402e7\" (UID: \"e79c8385-25d9-4a8e-a552-fce14c2402e7\") " Oct 30 19:01:04.111321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.111298 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca" pod "e79c8385-25d9-4a8e-a552-fce14c2402e7" (UID: "e79c8385-25d9-4a8e-a552-fce14c2402e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:04.112873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.112849 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca" pod "e79c8385-25d9-4a8e-a552-fce14c2402e7" (UID: "e79c8385-25d9-4a8e-a552-fce14c2402e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:04.112959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.112907 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m" pod "e79c8385-25d9-4a8e-a552-fce14c2402e7" (UID: "e79c8385-25d9-4a8e-a552-fce14c2402e7"). InnerVolumeSpecName "kube-api-access-zhw6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:04.212492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.212464 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:04.212492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.212490 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhw6m\" (UniqueName: \"kubernetes.io/projected/e79c8385-25d9-4a8e-a552-fce14c2402e7-kube-api-access-zhw6m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:04.212492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.212500 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79c8385-25d9-4a8e-a552-fce14c2402e7-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:04.919588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.919552 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" event={"ID":"e79c8385-25d9-4a8e-a552-fce14c2402e7","Type":"ContainerDied","Data":"f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6"} Oct 30 19:01:04.919588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.919576 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh" Oct 30 19:01:04.919846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:04.919602 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9688141a5e5a82e15713372bfc114a3fd609ec73e038178621a4d36a1225cf6" Oct 30 19:01:07.852386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:07.852350 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s"] Oct 30 19:01:07.899070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:07.899029 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e7b96f88-drb6s"] Oct 30 19:01:08.833820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:08.833785 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml"] Oct 30 19:01:08.865931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:08.865885 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" podUID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" containerName="payload-processing" containerID="cri-o://413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a" gracePeriod=25 Oct 30 19:01:08.880158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:08.880135 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ddb8b1dd-mzvml"] Oct 30 19:01:09.120946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.120912 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:01:09.154618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.154585 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\" (UniqueName: \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\") pod \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " Oct 30 19:01:09.154820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.154684 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\") pod \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\" (UID: \"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb\") " Oct 30 19:01:09.155044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.155020 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume" pod "6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" (UID: "6eaa9b9c-85b3-4d88-851d-1ef91c32eabb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:09.156861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.156833 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p" pod "6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" (UID: "6eaa9b9c-85b3-4d88-851d-1ef91c32eabb"). InnerVolumeSpecName "kube-api-access-g2f2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:09.256164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.256122 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2f2p\" (UniqueName: \"kubernetes.io/projected/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-kube-api-access-g2f2p\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:09.256164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.256164 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:09.667466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.667421 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171f7205-60d7-41e9-b344-0453cbc3567d" path="/var/lib/kubelet/pods/171f7205-60d7-41e9-b344-0453cbc3567d/volumes" Oct 30 19:01:09.667888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.667868 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3cb466-8629-4b1e-81c9-518264443bc3" path="/var/lib/kubelet/pods/8c3cb466-8629-4b1e-81c9-518264443bc3/volumes" Oct 30 19:01:09.953369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.953333 4789 generic.go:356] "Generic (PLEG): container finished" podID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" containerID="413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a" exitCode=0 Oct 30 19:01:09.953783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.953380 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" event={"ID":"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb","Type":"ContainerDied","Data":"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a"} Oct 30 19:01:09.953783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.953386 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" Oct 30 19:01:09.953783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.953421 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl" event={"ID":"6eaa9b9c-85b3-4d88-851d-1ef91c32eabb","Type":"ContainerDied","Data":"347d469ec0a8455aafcbb09989ac001544b0ff3cf3cbda95c87e5f990f4f883a"} Oct 30 19:01:09.953783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.953443 4789 scope.go:122] "RemoveContainer" containerID="413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a" Oct 30 19:01:09.962259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.962237 4789 scope.go:122] "RemoveContainer" containerID="413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a" Oct 30 19:01:09.962540 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:01:09.962518 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a\": container with ID starting with 413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a not found: ID does not exist" containerID="413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a" Oct 30 19:01:09.962598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:09.962548 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a"} err="failed to get container status \"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a\": rpc error: code = NotFound desc = could not find container \"413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a\": container with ID starting with 413a8be2e3e896122a7021af98c35a5bd0cbf3582be2ed4edb0439ed93692d2a not found: ID does not exist" Oct 30 19:01:10.108477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:10.108443 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:01:10.154827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:10.154783 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ddb8b1dd-7b5f46756b-rnvbl"] Oct 30 19:01:11.667840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:11.667803 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" path="/var/lib/kubelet/pods/6eaa9b9c-85b3-4d88-851d-1ef91c32eabb/volumes" Oct 30 19:01:12.608032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:12.607994 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:01:17.609280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.609200 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" podUID="0a31f850-0125-42c1-8ddb-33bade562dc2" containerName="payload-processing" containerID="cri-o://f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde" gracePeriod=25 Oct 30 19:01:17.892841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.892815 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:01:17.996594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.996557 4789 generic.go:356] "Generic (PLEG): container finished" podID="0a31f850-0125-42c1-8ddb-33bade562dc2" containerID="f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde" exitCode=0 Oct 30 19:01:17.996749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.996604 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" Oct 30 19:01:17.996749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.996636 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" event={"ID":"0a31f850-0125-42c1-8ddb-33bade562dc2","Type":"ContainerDied","Data":"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde"} Oct 30 19:01:17.996749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.996692 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc" event={"ID":"0a31f850-0125-42c1-8ddb-33bade562dc2","Type":"ContainerDied","Data":"4d721bde7f7a036fdada65f6435ed69d5b7bb2fbc4efcc69922e821a34e00776"} Oct 30 19:01:17.996749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:17.996717 4789 scope.go:122] "RemoveContainer" containerID="f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde" Oct 30 19:01:18.006735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.006716 4789 scope.go:122] "RemoveContainer" containerID="f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde" Oct 30 19:01:18.007259 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:01:18.007228 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde\": container with ID starting with f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde not found: ID does not exist" containerID="f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde" Oct 30 19:01:18.007368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.007272 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde"} err="failed to get container status \"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde\": rpc error: code = NotFound desc = could not find container \"f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde\": container with ID starting with f0e38fb35fef8f38c082949963be761048531e0fa92cca6fef2ab96a505f0dde not found: ID does not exist" Oct 30 19:01:18.039581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.039548 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\" (UniqueName: \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\") pod \"0a31f850-0125-42c1-8ddb-33bade562dc2\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " Oct 30 19:01:18.039755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.039646 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\") pod \"0a31f850-0125-42c1-8ddb-33bade562dc2\" (UID: \"0a31f850-0125-42c1-8ddb-33bade562dc2\") " Oct 30 19:01:18.040046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.040008 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume" pod "0a31f850-0125-42c1-8ddb-33bade562dc2" (UID: "0a31f850-0125-42c1-8ddb-33bade562dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:18.041646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.041619 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr" pod "0a31f850-0125-42c1-8ddb-33bade562dc2" (UID: "0a31f850-0125-42c1-8ddb-33bade562dc2"). InnerVolumeSpecName "kube-api-access-jkndr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:18.141139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.141056 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkndr\" (UniqueName: \"kubernetes.io/projected/0a31f850-0125-42c1-8ddb-33bade562dc2-kube-api-access-jkndr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:18.141139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.141087 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a31f850-0125-42c1-8ddb-33bade562dc2-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:18.197555 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:01:18.197513 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:01:18.412314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.412225 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:01:18.416011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:18.415983 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df563ff7-6cc769b5bd-hm7xc"] Oct 30 19:01:19.539825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:19.539789 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh"] Oct 30 19:01:19.584805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:19.584765 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df563ff7-lh4vh"] Oct 30 19:01:19.666550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:19.666520 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a31f850-0125-42c1-8ddb-33bade562dc2" path="/var/lib/kubelet/pods/0a31f850-0125-42c1-8ddb-33bade562dc2/volumes" Oct 30 19:01:19.666920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:01:19.666896 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79c8385-25d9-4a8e-a552-fce14c2402e7" path="/var/lib/kubelet/pods/e79c8385-25d9-4a8e-a552-fce14c2402e7/volumes" Oct 30 19:01:48.195774 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:01:48.195693 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:02:01.348594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.348560 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:02:01.349153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349134 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0a31f850-0125-42c1-8ddb-33bade562dc2" containerName="payload-processing" Oct 30 19:02:01.349256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349156 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a31f850-0125-42c1-8ddb-33bade562dc2" containerName="payload-processing" Oct 30 19:02:01.349256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349171 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e79c8385-25d9-4a8e-a552-fce14c2402e7" containerName="revoke-keys" Oct 30 19:02:01.349256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349178 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79c8385-25d9-4a8e-a552-fce14c2402e7" containerName="revoke-keys" Oct 30 19:02:01.349256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349231 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" containerName="payload-processing" Oct 30 19:02:01.349256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349240 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" containerName="payload-processing" Oct 30 19:02:01.349510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349356 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a31f850-0125-42c1-8ddb-33bade562dc2" containerName="payload-processing" Oct 30 19:02:01.349510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349367 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79c8385-25d9-4a8e-a552-fce14c2402e7" containerName="revoke-keys" Oct 30 19:02:01.349510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.349414 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eaa9b9c-85b3-4d88-851d-1ef91c32eabb" containerName="payload-processing" Oct 30 19:02:01.352931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.352911 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.402250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.402221 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9275c3\"/\"simulator-w1-9275c3-kserve-self-signed-certs\"" Oct 30 19:02:01.429163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429135 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shs6\" (UniqueName: \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.429273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429170 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.429273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429189 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.429366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429271 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.429402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429362 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.429402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.429391 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.448538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.448518 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9275c3\"/\"default-dockercfg-7g5c2\"" Oct 30 19:02:01.448731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.448603 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9275c3\"/\"openshift-service-ca.crt\"" Oct 30 19:02:01.448731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.448578 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9275c3\"/\"kube-root-ca.crt\"" Oct 30 19:02:01.451358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.451335 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:02:01.530222 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530183 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shs6\" (UniqueName: \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530250 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530282 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530322 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530359 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530395 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530745 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530815 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.530938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.530899 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.532747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.532723 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.533065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.533045 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.590903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.590859 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shs6\" (UniqueName: \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\") pod \"simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:01.664421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:01.664335 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:02.040418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.040381 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:02:02.047268 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:02:02.047241 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a3b803_e7fc_4322_aaa4_a8cc0e725cf7.slice/crio-f843c163b450cb48f99f7d250fd5fc1a051d765f7ba47d939ee5a7e24df1fdbe WatchSource:0}: Error finding container f843c163b450cb48f99f7d250fd5fc1a051d765f7ba47d939ee5a7e24df1fdbe: Status 404 returned error can't find the container with id f843c163b450cb48f99f7d250fd5fc1a051d765f7ba47d939ee5a7e24df1fdbe Oct 30 19:02:02.222546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.222509 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" event={"ID":"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7","Type":"ContainerStarted","Data":"f843c163b450cb48f99f7d250fd5fc1a051d765f7ba47d939ee5a7e24df1fdbe"} Oct 30 19:02:02.641975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.641940 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:02:02.647344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.647323 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.706677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.706632 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:02:02.742820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.742784 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5qz\" (UniqueName: \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.742998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.742843 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.745433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.745408 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-2301cb\"" Oct 30 19:02:02.745546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.745453 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-2301cb-dockercfg-tvjxz\"" Oct 30 19:02:02.843744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.843715 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5qz\" (UniqueName: \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.843925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.843765 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.844495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.844476 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.903597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.901955 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5qz\" (UniqueName: \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\") pod \"payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:02.960174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:02.960139 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:03.341692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:03.341655 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:02:03.349533 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:02:03.349501 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf0c815_d37d_4f09_8e4b_1fbd65d5dd7f.slice/crio-09d744add1ac816bf638580c3ae0a51ce9f44eabd3c5cb7c15139b428c7d47ab WatchSource:0}: Error finding container 09d744add1ac816bf638580c3ae0a51ce9f44eabd3c5cb7c15139b428c7d47ab: Status 404 returned error can't find the container with id 09d744add1ac816bf638580c3ae0a51ce9f44eabd3c5cb7c15139b428c7d47ab Oct 30 19:02:04.236857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:04.236816 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" event={"ID":"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f","Type":"ContainerStarted","Data":"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3"} Oct 30 19:02:04.237369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:04.236928 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:04.237369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:04.236954 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" event={"ID":"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f","Type":"ContainerStarted","Data":"09d744add1ac816bf638580c3ae0a51ce9f44eabd3c5cb7c15139b428c7d47ab"} Oct 30 19:02:04.347613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:04.347572 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" podStartSLOduration=2.347558745 podStartE2EDuration="2.347558745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:04.346042349 +0000 UTC m=+960.847952834" watchObservedRunningTime="2026-10-30 19:02:04.347558745 +0000 UTC m=+960.849469231" Oct 30 19:02:05.244631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:05.244590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" event={"ID":"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7","Type":"ContainerStarted","Data":"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313"} Oct 30 19:02:05.354949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:05.354899 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" podStartSLOduration=2.101300685 podStartE2EDuration="4.354886655s" totalImagesPullingTime="2.25358597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:05.354526576 +0000 UTC m=+961.856437064" watchObservedRunningTime="2026-10-30 19:02:05.354886655 +0000 UTC m=+961.856797141" Oct 30 19:02:05.666967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:05.666897 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:11.667646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:11.667609 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:11.679447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:11.679424 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:12.289082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:12.289054 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:02:15.248413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:15.248389 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:02:18.195603 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:02:18.195552 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:02:23.833926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.833892 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:02:23.837464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.837446 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:23.886548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.886515 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-22fb18\"/\"kube-root-ca.crt\"" Oct 30 19:02:23.900286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.900261 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-22fb18\"/\"openshift-service-ca.crt\"" Oct 30 19:02:23.932806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.932774 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-22fb18\"/\"default-dockercfg-tj7q4\"" Oct 30 19:02:23.932965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.932820 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-22fb18\"/\"simulator-w3-22fb18-kserve-self-signed-certs\"" Oct 30 19:02:23.934599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:23.934574 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:02:24.022865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.022833 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.022865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.022869 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.023104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.022904 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.023104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.022998 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.023104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.023027 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd864\" (UniqueName: \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.023104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.023098 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.123987 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124039 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124069 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd864\" (UniqueName: \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124115 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124239 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124278 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124628 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124732 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.124839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.124793 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.126864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.126838 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.126995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.126952 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.182386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.182350 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd864\" (UniqueName: \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\") pod \"simulator-w3-22fb18-kserve-74f986f68f-c6l5r\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.449653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.449624 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:24.821879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:24.821789 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:02:24.828921 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:02:24.828891 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ca16be_6cf0_4f92_af5c_4b515d7e7f27.slice/crio-fe2146e6d092fa34a492a6e5b1f35a02d393e07e4cd596d3166c89af26a03e26 WatchSource:0}: Error finding container fe2146e6d092fa34a492a6e5b1f35a02d393e07e4cd596d3166c89af26a03e26: Status 404 returned error can't find the container with id fe2146e6d092fa34a492a6e5b1f35a02d393e07e4cd596d3166c89af26a03e26 Oct 30 19:02:25.353144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:25.353106 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" event={"ID":"31ca16be-6cf0-4f92-af5c-4b515d7e7f27","Type":"ContainerStarted","Data":"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd"} Oct 30 19:02:25.353144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:25.353150 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" event={"ID":"31ca16be-6cf0-4f92-af5c-4b515d7e7f27","Type":"ContainerStarted","Data":"fe2146e6d092fa34a492a6e5b1f35a02d393e07e4cd596d3166c89af26a03e26"} Oct 30 19:02:25.450489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:25.450456 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:25.465139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:25.465083 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podStartSLOduration=2.465069319 podStartE2EDuration="2.465069319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:25.463675437 +0000 UTC m=+981.965585924" watchObservedRunningTime="2026-10-30 19:02:25.465069319 +0000 UTC m=+981.966979804" Oct 30 19:02:29.959367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.959332 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 19:02:29.963437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.963416 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975448 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975488 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcprk\" (UniqueName: \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975510 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975571 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975670 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:29.975718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:29.975701 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.059153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.059120 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"openshift-service-ca.crt\"" Oct 30 19:02:30.059355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.059164 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"simulator-w4-1323a7-kserve-self-signed-certs\"" Oct 30 19:02:30.059355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.059172 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"kube-root-ca.crt\"" Oct 30 19:02:30.059524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.059498 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"default-dockercfg-rrhz5\"" Oct 30 19:02:30.060987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.060967 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 19:02:30.076767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076745 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.076881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076778 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.076881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076846 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.076881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076866 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcprk\" (UniqueName: \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.077058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076891 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.077058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.076917 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.077294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.077274 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.077378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.077296 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.077541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.077525 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.079547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.079524 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.079651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.079570 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.135027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.134988 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcprk\" (UniqueName: \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\") pod \"simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.275129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.275042 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:30.663310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:30.663203 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 19:02:30.669360 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:02:30.669335 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29dbf484_abcb_4246_8e97_9784359be27c.slice/crio-333dca9bcf451e244c9e60511034e17570e6fbdf8b5d90ba429880b849d250e6 WatchSource:0}: Error finding container 333dca9bcf451e244c9e60511034e17570e6fbdf8b5d90ba429880b849d250e6: Status 404 returned error can't find the container with id 333dca9bcf451e244c9e60511034e17570e6fbdf8b5d90ba429880b849d250e6 Oct 30 19:02:31.384737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:31.384702 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" event={"ID":"29dbf484-abcb-4246-8e97-9784359be27c","Type":"ContainerStarted","Data":"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718"} Oct 30 19:02:31.385148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:31.384749 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" event={"ID":"29dbf484-abcb-4246-8e97-9784359be27c","Type":"ContainerStarted","Data":"333dca9bcf451e244c9e60511034e17570e6fbdf8b5d90ba429880b849d250e6"} Oct 30 19:02:31.498592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:31.498537 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" podStartSLOduration=2.498521783 podStartE2EDuration="2.498521783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:31.496912046 +0000 UTC m=+987.998822532" watchObservedRunningTime="2026-10-30 19:02:31.498521783 +0000 UTC m=+988.000432269" Oct 30 19:02:32.275409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:32.275371 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:34.450524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:34.450487 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:34.463339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:34.463311 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:35.424076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:35.424046 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:02:40.276125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:40.276079 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:40.289352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:40.289324 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:40.449812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:40.449786 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 19:02:48.196143 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:02:48.196106 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:02:57.764353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.764318 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:02:57.767061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.767039 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826574 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxtx\" (UniqueName: \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826616 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826689 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826732 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826844 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.826889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.826885 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.828965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.828941 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9275c3\"/\"premium-w1-9275c3-kserve-self-signed-certs\"" Oct 30 19:02:57.864076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.864049 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:02:57.928279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928244 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxtx\" (UniqueName: \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928280 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928301 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928335 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928396 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928437 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928740 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928757 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.928868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.928802 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.930662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.930641 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.930925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.930907 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:57.988330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:57.988305 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxtx\" (UniqueName: \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\") pod \"premium-w1-9275c3-kserve-5899cb8d4b-jr5b5\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:58.078426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:58.078345 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:02:58.450516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:58.450484 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:02:58.457011 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:02:58.456983 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ca60daf_a031_47e5_93a5_57ac0d607a36.slice/crio-da2bbbd082358b290510ede2dffda65363f5cbc383b78d1b115631bf587854c2 WatchSource:0}: Error finding container da2bbbd082358b290510ede2dffda65363f5cbc383b78d1b115631bf587854c2: Status 404 returned error can't find the container with id da2bbbd082358b290510ede2dffda65363f5cbc383b78d1b115631bf587854c2 Oct 30 19:02:58.532349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:58.532319 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" event={"ID":"8ca60daf-a031-47e5-93a5-57ac0d607a36","Type":"ContainerStarted","Data":"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b"} Oct 30 19:02:58.532496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:58.532369 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" event={"ID":"8ca60daf-a031-47e5-93a5-57ac0d607a36","Type":"ContainerStarted","Data":"da2bbbd082358b290510ede2dffda65363f5cbc383b78d1b115631bf587854c2"} Oct 30 19:02:58.644252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:58.644183 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podStartSLOduration=1.6441683 podStartE2EDuration="1.6441683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:58.642807111 +0000 UTC m=+1015.144717594" watchObservedRunningTime="2026-10-30 19:02:58.6441683 +0000 UTC m=+1015.146078786" Oct 30 19:02:59.078446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:02:59.078406 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:03:08.079302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:08.079265 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:03:08.091650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:08.091619 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:03:08.601376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:08.601346 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:03:16.822610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.822513 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 19:03:16.826677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.826649 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.875910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.875880 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-330025\"/\"openshift-service-ca.crt\"" Oct 30 19:03:16.905635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905592 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8xv\" (UniqueName: \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.905802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905707 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.905802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905733 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.905942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905808 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.905942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905856 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.905942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.905935 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:16.921912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.921877 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-330025\"/\"kube-root-ca.crt\"" Oct 30 19:03:16.922078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.921931 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-330025\"/\"simulator-w2-330025-kserve-self-signed-certs\"" Oct 30 19:03:16.922078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.921958 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-330025\"/\"default-dockercfg-77fzp\"" Oct 30 19:03:16.924145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:16.924125 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 19:03:17.006723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006674 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.006966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006781 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8xv\" (UniqueName: \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.006966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006847 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.006966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006871 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.006966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006895 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.006966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.006951 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.007295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.007196 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.007295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.007250 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.007638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.007616 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.010125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.010102 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.010337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.010261 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.068541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.068504 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8xv\" (UniqueName: \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\") pod \"simulator-w2-330025-kserve-c5c76cf84-pmskk\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.139138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.138374 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:17.537548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.537508 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 19:03:17.543932 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:03:17.543905 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d69644f_e496_43f7_aa37_6264364d8078.slice/crio-d2a04fd931ad850660e737be035184e7379510417794576913cbc6b0cfc9a018 WatchSource:0}: Error finding container d2a04fd931ad850660e737be035184e7379510417794576913cbc6b0cfc9a018: Status 404 returned error can't find the container with id d2a04fd931ad850660e737be035184e7379510417794576913cbc6b0cfc9a018 Oct 30 19:03:17.640029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.639994 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" event={"ID":"8d69644f-e496-43f7-aa37-6264364d8078","Type":"ContainerStarted","Data":"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0"} Oct 30 19:03:17.640187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.640039 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" event={"ID":"8d69644f-e496-43f7-aa37-6264364d8078","Type":"ContainerStarted","Data":"d2a04fd931ad850660e737be035184e7379510417794576913cbc6b0cfc9a018"} Oct 30 19:03:17.752022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:17.751964 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podStartSLOduration=1.751948874 podStartE2EDuration="1.751948874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:17.750605135 +0000 UTC m=+1034.252515622" watchObservedRunningTime="2026-10-30 19:03:17.751948874 +0000 UTC m=+1034.253859404" Oct 30 19:03:18.139266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:18.139232 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:18.196082 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:03:18.196013 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:03:27.139050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:27.139013 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:27.151678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:27.151651 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:27.702344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:27.702312 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:03:38.535545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.535508 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 19:03:38.539321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.539299 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620349 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b289\" (UniqueName: \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620397 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620492 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620516 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620531 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.620676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.620646 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.634498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.634471 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"premium-w4-1323a7-kserve-self-signed-certs\"" Oct 30 19:03:38.636786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.636751 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 19:03:38.721731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.721696 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.721917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.721736 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.721917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.721762 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.721917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.721882 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.722094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.722011 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b289\" (UniqueName: \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.722094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.722082 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.722205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.722148 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.722301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.722237 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.722364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.722303 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.724867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.724841 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.724867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.724863 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.780478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.780442 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b289\" (UniqueName: \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\") pod \"premium-w4-1323a7-kserve-8445798475-hcpdp\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:38.852190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:38.852099 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:39.241945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:39.241905 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 19:03:39.248803 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:03:39.248775 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8575aa9_12e3_474c_b85f_e14817c4d6ac.slice/crio-45141f25906f5da2779406ed6146bfd9728c861b37593cc682c980f06336f457 WatchSource:0}: Error finding container 45141f25906f5da2779406ed6146bfd9728c861b37593cc682c980f06336f457: Status 404 returned error can't find the container with id 45141f25906f5da2779406ed6146bfd9728c861b37593cc682c980f06336f457 Oct 30 19:03:39.757850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:39.757242 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" event={"ID":"a8575aa9-12e3-474c-b85f-e14817c4d6ac","Type":"ContainerStarted","Data":"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a"} Oct 30 19:03:39.757850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:39.757290 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" event={"ID":"a8575aa9-12e3-474c-b85f-e14817c4d6ac","Type":"ContainerStarted","Data":"45141f25906f5da2779406ed6146bfd9728c861b37593cc682c980f06336f457"} Oct 30 19:03:39.853019 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:39.852987 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:39.868683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:39.868630 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" podStartSLOduration=1.86861613 podStartE2EDuration="1.86861613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:39.86797812 +0000 UTC m=+1056.369888608" watchObservedRunningTime="2026-10-30 19:03:39.86861613 +0000 UTC m=+1056.370526615" Oct 30 19:03:48.197059 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:03:48.197015 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:03:48.853027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:48.852994 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:48.865640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:48.865615 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:03:49.824899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:03:49.824859 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 19:04:07.183442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.183406 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rzr4l-wkkhj"] Oct 30 19:04:07.186988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.186969 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.241383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.241352 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rzr4l-wkkhj"] Oct 30 19:04:07.289507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.289482 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.289638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.289539 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.289638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.289621 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.289747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.289727 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6bs\" (UniqueName: \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.390646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.390617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.390815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.390653 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.390815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.390697 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6bs\" (UniqueName: \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.390815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.390717 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.391100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.391077 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.391322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.391297 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.391624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.391601 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.448558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.448529 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6bs\" (UniqueName: \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\") pod \"periodic-gathering-rzr4l-wkkhj\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.499774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.499735 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:04:07.877487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.877401 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rzr4l-wkkhj"] Oct 30 19:04:07.884384 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:04:07.884357 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b25566_df8b_4147_8bd3_1382cf94587f.slice/crio-1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff WatchSource:0}: Error finding container 1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff: Status 404 returned error can't find the container with id 1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff Oct 30 19:04:07.915272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:07.915240 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" event={"ID":"e3b25566-df8b-4147-8bd3-1382cf94587f","Type":"ContainerStarted","Data":"1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff"} Oct 30 19:04:08.923040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:08.923000 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" event={"ID":"e3b25566-df8b-4147-8bd3-1382cf94587f","Type":"ContainerStarted","Data":"6c8f2958d903c378886a66ace22db80ba2a024d63c2a491950722e914787b957"} Oct 30 19:04:09.033714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:09.033662 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" podStartSLOduration=2.033648694 podStartE2EDuration="2.033648694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:09.032093438 +0000 UTC m=+1085.534003925" watchObservedRunningTime="2026-10-30 19:04:09.033648694 +0000 UTC m=+1085.535559179" Oct 30 19:04:17.973955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:17.973918 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr"] Oct 30 19:04:17.977876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:17.977854 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.027881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.027848 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:18.027881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.027861 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:04:18.074354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.074325 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:04:18.074509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.074413 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:04:18.080345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.079080 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr"] Oct 30 19:04:18.101461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.101427 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnmt\" (UniqueName: \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.101461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.101463 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.101700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.101633 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.196065 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:04:18.196021 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:04:18.202647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.202621 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.202739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.202715 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnmt\" (UniqueName: \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.202788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.202744 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.203500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.203481 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.205357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.205329 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.262033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.261954 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnmt\" (UniqueName: \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\") pod \"maas-api-revoke-sub-450333b6-xvhxr\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.290778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.290747 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:18.671225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.671126 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr"] Oct 30 19:04:18.677977 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:04:18.677949 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ca318c_08ec_4967_9574_c287bea4f15a.slice/crio-f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b WatchSource:0}: Error finding container f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b: Status 404 returned error can't find the container with id f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b Oct 30 19:04:18.975545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:18.975506 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" event={"ID":"49ca318c-08ec-4967-9574-c287bea4f15a","Type":"ContainerStarted","Data":"f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b"} Oct 30 19:04:19.985137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:19.982964 4789 generic.go:356] "Generic (PLEG): container finished" podID="49ca318c-08ec-4967-9574-c287bea4f15a" containerID="5e765ce212499a4243ca23624826ed58c2389bdd7e55ace14648d89cd66becce" exitCode=0 Oct 30 19:04:19.985137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:19.983179 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" event={"ID":"49ca318c-08ec-4967-9574-c287bea4f15a","Type":"ContainerDied","Data":"5e765ce212499a4243ca23624826ed58c2389bdd7e55ace14648d89cd66becce"} Oct 30 19:04:21.129969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.129945 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:21.236018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.235981 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\") pod \"49ca318c-08ec-4967-9574-c287bea4f15a\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " Oct 30 19:04:21.236245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.236148 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\") pod \"49ca318c-08ec-4967-9574-c287bea4f15a\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " Oct 30 19:04:21.236245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.236200 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\" (UniqueName: \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\") pod \"49ca318c-08ec-4967-9574-c287bea4f15a\" (UID: \"49ca318c-08ec-4967-9574-c287bea4f15a\") " Oct 30 19:04:21.236470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.236438 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca" pod "49ca318c-08ec-4967-9574-c287bea4f15a" (UID: "49ca318c-08ec-4967-9574-c287bea4f15a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:21.236560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.236542 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:21.238214 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.238188 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca" pod "49ca318c-08ec-4967-9574-c287bea4f15a" (UID: "49ca318c-08ec-4967-9574-c287bea4f15a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:21.238348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.238331 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt" pod "49ca318c-08ec-4967-9574-c287bea4f15a" (UID: "49ca318c-08ec-4967-9574-c287bea4f15a"). InnerVolumeSpecName "kube-api-access-vdnmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:21.337068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.337034 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdnmt\" (UniqueName: \"kubernetes.io/projected/49ca318c-08ec-4967-9574-c287bea4f15a-kube-api-access-vdnmt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:21.337068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.337065 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49ca318c-08ec-4967-9574-c287bea4f15a-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:21.995616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.995580 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" event={"ID":"49ca318c-08ec-4967-9574-c287bea4f15a","Type":"ContainerDied","Data":"f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b"} Oct 30 19:04:21.995616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.995607 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr" Oct 30 19:04:21.995872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:21.995629 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c4de73c46e581708f4e7534e697773caf31a3acd84f10e101029552c17635b" Oct 30 19:04:35.921446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.921410 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql"] Oct 30 19:04:35.922571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.922545 4789 cpu_manager.go:414] "containerMap: removing container" podUID="49ca318c-08ec-4967-9574-c287bea4f15a" containerName="revoke-keys" Oct 30 19:04:35.922721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.922708 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ca318c-08ec-4967-9574-c287bea4f15a" containerName="revoke-keys" Oct 30 19:04:35.922950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.922937 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ca318c-08ec-4967-9574-c287bea4f15a" containerName="revoke-keys" Oct 30 19:04:35.926761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.926731 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:35.980103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:35.980067 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql"] Oct 30 19:04:36.025431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.024528 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:04:36.025431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.024597 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:04:36.025431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.024732 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:04:36.025431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.024792 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:36.076300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.076265 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.076493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.076307 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjb7q\" (UniqueName: \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.076493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.076381 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.177669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.177597 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.177669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.177632 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjb7q\" (UniqueName: \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.177669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.177652 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.178247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.178205 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.179994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.179968 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.242664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.242631 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjb7q\" (UniqueName: \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\") pod \"maas-api-revoke-sub-0e493f7c-d6cql\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.246261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.246235 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:36.680398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:36.680358 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql"] Oct 30 19:04:37.085662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:37.085628 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" event={"ID":"e1263749-9956-4e10-8dab-914a832e2040","Type":"ContainerStarted","Data":"85a37df6e81020b84899ad3e5ae2911a536448d1d3aa390e96fba8ef0fc83f6a"} Oct 30 19:04:38.093934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:38.093849 4789 generic.go:356] "Generic (PLEG): container finished" podID="e1263749-9956-4e10-8dab-914a832e2040" containerID="1174bb28454ee4404f5b85e1d060f20c798e278027babfcd849ed45fdfc1740b" exitCode=0 Oct 30 19:04:38.093934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:38.093888 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" event={"ID":"e1263749-9956-4e10-8dab-914a832e2040","Type":"ContainerDied","Data":"1174bb28454ee4404f5b85e1d060f20c798e278027babfcd849ed45fdfc1740b"} Oct 30 19:04:39.237115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.237089 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:39.330180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.330139 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\" (UniqueName: \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\") pod \"e1263749-9956-4e10-8dab-914a832e2040\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " Oct 30 19:04:39.330180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.330185 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\") pod \"e1263749-9956-4e10-8dab-914a832e2040\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " Oct 30 19:04:39.330432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.330320 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\") pod \"e1263749-9956-4e10-8dab-914a832e2040\" (UID: \"e1263749-9956-4e10-8dab-914a832e2040\") " Oct 30 19:04:39.330703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.330674 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca" pod "e1263749-9956-4e10-8dab-914a832e2040" (UID: "e1263749-9956-4e10-8dab-914a832e2040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:39.332317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.332293 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca" pod "e1263749-9956-4e10-8dab-914a832e2040" (UID: "e1263749-9956-4e10-8dab-914a832e2040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:39.332409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.332352 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q" pod "e1263749-9956-4e10-8dab-914a832e2040" (UID: "e1263749-9956-4e10-8dab-914a832e2040"). InnerVolumeSpecName "kube-api-access-bjb7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:39.431197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.431107 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1263749-9956-4e10-8dab-914a832e2040-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.431197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.431138 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjb7q\" (UniqueName: \"kubernetes.io/projected/e1263749-9956-4e10-8dab-914a832e2040-kube-api-access-bjb7q\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:39.431197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:39.431150 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1263749-9956-4e10-8dab-914a832e2040-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:40.107470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:40.107431 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" Oct 30 19:04:40.107649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:40.107476 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql" event={"ID":"e1263749-9956-4e10-8dab-914a832e2040","Type":"ContainerDied","Data":"85a37df6e81020b84899ad3e5ae2911a536448d1d3aa390e96fba8ef0fc83f6a"} Oct 30 19:04:40.107649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:40.107523 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a37df6e81020b84899ad3e5ae2911a536448d1d3aa390e96fba8ef0fc83f6a" Oct 30 19:04:48.195329 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:04:48.195252 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:04:48.835370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.835339 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2"] Oct 30 19:04:48.835779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.835766 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e1263749-9956-4e10-8dab-914a832e2040" containerName="revoke-keys" Oct 30 19:04:48.835820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.835780 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1263749-9956-4e10-8dab-914a832e2040" containerName="revoke-keys" Oct 30 19:04:48.835855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.835849 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1263749-9956-4e10-8dab-914a832e2040" containerName="revoke-keys" Oct 30 19:04:48.838920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.838898 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:48.888031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.888003 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:04:48.927481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.927446 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrl8\" (UniqueName: \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:48.927645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.927511 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:48.927645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.927616 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:48.934009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.933989 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:48.934105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.934010 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:04:48.934105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.934050 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:04:48.936134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:48.936116 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2"] Oct 30 19:04:49.028435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.028406 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrl8\" (UniqueName: \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.028716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.028461 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.028716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.028508 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.029204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.029178 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.030831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.030813 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.086329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.086274 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrl8\" (UniqueName: \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\") pod \"maas-api-revoke-sub-e1b57173-qdtj2\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.152041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.152019 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:49.524876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:49.524845 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2"] Oct 30 19:04:49.531461 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:04:49.531418 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d584b_bb01_4f4c_9da0_b60d115bdbc1.slice/crio-66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9 WatchSource:0}: Error finding container 66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9: Status 404 returned error can't find the container with id 66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9 Oct 30 19:04:50.165709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:50.165620 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" event={"ID":"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1","Type":"ContainerStarted","Data":"b9affdc8dd65cbbfc91678d893879d1e5d671140e4f19b2db5c32e9f8a9eeeaa"} Oct 30 19:04:50.165853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:50.165718 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" event={"ID":"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1","Type":"ContainerStarted","Data":"66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9"} Oct 30 19:04:50.275673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:50.275622 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" podStartSLOduration=2.275608545 podStartE2EDuration="2.275608545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:50.275065371 +0000 UTC m=+1126.776975858" watchObservedRunningTime="2026-10-30 19:04:50.275608545 +0000 UTC m=+1126.777519030" Oct 30 19:04:51.173188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:51.173155 4789 generic.go:356] "Generic (PLEG): container finished" podID="bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" containerID="b9affdc8dd65cbbfc91678d893879d1e5d671140e4f19b2db5c32e9f8a9eeeaa" exitCode=0 Oct 30 19:04:51.173624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:51.173250 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" event={"ID":"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1","Type":"ContainerDied","Data":"b9affdc8dd65cbbfc91678d893879d1e5d671140e4f19b2db5c32e9f8a9eeeaa"} Oct 30 19:04:52.312871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.312845 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:52.463402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.463372 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\" (UniqueName: \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\") pod \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " Oct 30 19:04:52.463586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.463492 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\") pod \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " Oct 30 19:04:52.463717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.463696 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\") pod \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\" (UID: \"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1\") " Oct 30 19:04:52.463895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.463872 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca" pod "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" (UID: "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:52.464085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.464068 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:52.465661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.465636 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8" pod "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" (UID: "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1"). InnerVolumeSpecName "kube-api-access-jtrl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:52.465661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.465647 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca" pod "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" (UID: "bb8d584b-bb01-4f4c-9da0-b60d115bdbc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:52.564978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.564936 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:52.564978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:52.564974 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrl8\" (UniqueName: \"kubernetes.io/projected/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1-kube-api-access-jtrl8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:53.185534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:53.185504 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" Oct 30 19:04:53.185708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:53.185536 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2" event={"ID":"bb8d584b-bb01-4f4c-9da0-b60d115bdbc1","Type":"ContainerDied","Data":"66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9"} Oct 30 19:04:53.185708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:53.185580 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66394ec116b903af554fa51079d67864e16f74b60d872cb08cdd562d003738c9" Oct 30 19:04:55.652510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.652466 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk"] Oct 30 19:04:55.653147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.653126 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" containerName="revoke-keys" Oct 30 19:04:55.653272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.653149 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" containerName="revoke-keys" Oct 30 19:04:55.653405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.653384 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" containerName="revoke-keys" Oct 30 19:04:55.656777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.656757 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.754981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.754935 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:55.754981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.754966 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:04:55.754981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.754937 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:04:55.755244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.755001 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:04:55.758434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.758406 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk"] Oct 30 19:04:55.798669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.798582 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.798813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.798706 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766rt\" (UniqueName: \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.798813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.798730 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.900182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.900146 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.900368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.900249 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766rt\" (UniqueName: \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.900368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.900272 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.900827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.900799 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.902500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.902479 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.958576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.958542 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766rt\" (UniqueName: \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\") pod \"maas-api-revoke-sub-4884b00b-k8pfk\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:55.972229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:55.972188 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:56.345060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:56.345029 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk"] Oct 30 19:04:56.351928 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:04:56.351895 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da0c03e_6ec9_47c6_aa85_3d49864d1a9b.slice/crio-bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35 WatchSource:0}: Error finding container bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35: Status 404 returned error can't find the container with id bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35 Oct 30 19:04:57.207608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:57.207578 4789 generic.go:356] "Generic (PLEG): container finished" podID="6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" containerID="4dd4ed65d104042708ffaca8910ab0a9eec5178d0f15c08014925bc8061e364b" exitCode=0 Oct 30 19:04:57.207926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:57.207655 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" event={"ID":"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b","Type":"ContainerDied","Data":"4dd4ed65d104042708ffaca8910ab0a9eec5178d0f15c08014925bc8061e364b"} Oct 30 19:04:57.207926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:57.207708 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" event={"ID":"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b","Type":"ContainerStarted","Data":"bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35"} Oct 30 19:04:58.367631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.367607 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:58.525296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.525198 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\") pod \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " Oct 30 19:04:58.525296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.525277 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\" (UniqueName: \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\") pod \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " Oct 30 19:04:58.525490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.525390 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\") pod \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\" (UID: \"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b\") " Oct 30 19:04:58.525793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.525757 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca" pod "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" (UID: "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:58.527389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.527366 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt" pod "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" (UID: "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b"). InnerVolumeSpecName "kube-api-access-766rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:58.527472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.527445 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca" pod "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" (UID: "6da0c03e-6ec9-47c6-aa85-3d49864d1a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:58.626043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.626006 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:58.626043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.626036 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766rt\" (UniqueName: \"kubernetes.io/projected/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-kube-api-access-766rt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:58.626239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:58.626050 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:59.219524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:59.219486 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" Oct 30 19:04:59.219710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:59.219486 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk" event={"ID":"6da0c03e-6ec9-47c6-aa85-3d49864d1a9b","Type":"ContainerDied","Data":"bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35"} Oct 30 19:04:59.219710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:04:59.219641 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd6d2f28afceed43c1106047de8cc26deade7037e81de1b3d54e687fdb535e35" Oct 30 19:05:11.223603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.223562 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k"] Oct 30 19:05:11.224487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.224223 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" containerName="revoke-keys" Oct 30 19:05:11.224487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.224244 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" containerName="revoke-keys" Oct 30 19:05:11.224487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.224379 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" containerName="revoke-keys" Oct 30 19:05:11.228741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.228721 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.244795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.244774 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.244903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.244808 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.244903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.244846 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjrj\" (UniqueName: \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.280025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.280002 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:05:11.326609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.326581 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:05:11.326779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.326633 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:11.326779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.326716 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:05:11.328906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.328879 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k"] Oct 30 19:05:11.346055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.346031 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.346148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.346106 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjrj\" (UniqueName: \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.346256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.346240 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.346715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.346696 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.348486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.348465 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.404242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.404197 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjrj\" (UniqueName: \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\") pod \"maas-api-revoke-sub-0290e228-dfx5k\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.541173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.541105 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:11.913972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:11.913889 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k"] Oct 30 19:05:11.921112 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:11.921089 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4146c77f_f264_4470_9b08_43caec4041b7.slice/crio-520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469 WatchSource:0}: Error finding container 520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469: Status 404 returned error can't find the container with id 520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469 Oct 30 19:05:12.295390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:12.295358 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" event={"ID":"4146c77f-f264-4470-9b08-43caec4041b7","Type":"ContainerStarted","Data":"39a7be136d9e5c0d9d44fd01b245bd1e20819c2008fe85cfffc76327aff01a16"} Oct 30 19:05:12.295852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:12.295422 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" event={"ID":"4146c77f-f264-4470-9b08-43caec4041b7","Type":"ContainerStarted","Data":"520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469"} Oct 30 19:05:12.406888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:12.406840 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" podStartSLOduration=1.406826648 podStartE2EDuration="1.406826648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:12.404909503 +0000 UTC m=+1148.906819989" watchObservedRunningTime="2026-10-30 19:05:12.406826648 +0000 UTC m=+1148.908737135" Oct 30 19:05:13.301683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:13.301649 4789 generic.go:356] "Generic (PLEG): container finished" podID="4146c77f-f264-4470-9b08-43caec4041b7" containerID="39a7be136d9e5c0d9d44fd01b245bd1e20819c2008fe85cfffc76327aff01a16" exitCode=0 Oct 30 19:05:13.302064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:13.301736 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" event={"ID":"4146c77f-f264-4470-9b08-43caec4041b7","Type":"ContainerDied","Data":"39a7be136d9e5c0d9d44fd01b245bd1e20819c2008fe85cfffc76327aff01a16"} Oct 30 19:05:14.450899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.450873 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:14.477982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.477955 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\" (UniqueName: \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\") pod \"4146c77f-f264-4470-9b08-43caec4041b7\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " Oct 30 19:05:14.478139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.478006 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\") pod \"4146c77f-f264-4470-9b08-43caec4041b7\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " Oct 30 19:05:14.478139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.478047 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\") pod \"4146c77f-f264-4470-9b08-43caec4041b7\" (UID: \"4146c77f-f264-4470-9b08-43caec4041b7\") " Oct 30 19:05:14.478413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.478387 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca" pod "4146c77f-f264-4470-9b08-43caec4041b7" (UID: "4146c77f-f264-4470-9b08-43caec4041b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:14.480079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.480057 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj" pod "4146c77f-f264-4470-9b08-43caec4041b7" (UID: "4146c77f-f264-4470-9b08-43caec4041b7"). InnerVolumeSpecName "kube-api-access-5wjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:14.480184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.480077 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca" pod "4146c77f-f264-4470-9b08-43caec4041b7" (UID: "4146c77f-f264-4470-9b08-43caec4041b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:14.579442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.579367 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.579442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.579397 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4146c77f-f264-4470-9b08-43caec4041b7-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.579442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:14.579411 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjrj\" (UniqueName: \"kubernetes.io/projected/4146c77f-f264-4470-9b08-43caec4041b7-kube-api-access-5wjrj\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:15.314342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:15.314296 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" Oct 30 19:05:15.314543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:15.314340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k" event={"ID":"4146c77f-f264-4470-9b08-43caec4041b7","Type":"ContainerDied","Data":"520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469"} Oct 30 19:05:15.314543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:15.314397 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="520dd7786500e238e8d17a4ff44af031ffcd5ee772755755bb369782f965c469" Oct 30 19:05:18.196570 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:05:18.196525 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:05:18.916939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.916906 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7"] Oct 30 19:05:18.917330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.917316 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4146c77f-f264-4470-9b08-43caec4041b7" containerName="revoke-keys" Oct 30 19:05:18.917399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.917331 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4146c77f-f264-4470-9b08-43caec4041b7" containerName="revoke-keys" Oct 30 19:05:18.917439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.917429 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4146c77f-f264-4470-9b08-43caec4041b7" containerName="revoke-keys" Oct 30 19:05:18.919514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.919488 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:18.968608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:18.968580 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:19.014847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.014818 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:05:19.014847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.014819 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:05:19.015094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.015076 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:05:19.017306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.017285 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7"] Oct 30 19:05:19.021573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.021551 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.021681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.021587 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.021745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.021719 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkrm\" (UniqueName: \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.123008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.122973 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.123192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.123013 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.123192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.123077 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkrm\" (UniqueName: \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.123647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.123626 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.125322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.125306 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.181597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.181536 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkrm\" (UniqueName: \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\") pod \"maas-api-revoke-sub-f7c0445b-ffbl7\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.231070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.231046 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:19.601980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:19.601944 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7"] Oct 30 19:05:19.608224 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:19.608184 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a528eea_eacd_49f4_8047_948171ab4b55.slice/crio-646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad WatchSource:0}: Error finding container 646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad: Status 404 returned error can't find the container with id 646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad Oct 30 19:05:20.348816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:20.348782 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" event={"ID":"0a528eea-eacd-49f4-8047-948171ab4b55","Type":"ContainerStarted","Data":"ffbd260bb4c6b631ff6750ac10fb6c86eac05a4ab7cb8dc2cda68297f1f6a7ba"} Oct 30 19:05:20.349311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:20.348834 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" event={"ID":"0a528eea-eacd-49f4-8047-948171ab4b55","Type":"ContainerStarted","Data":"646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad"} Oct 30 19:05:20.459896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:20.459831 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" podStartSLOduration=2.459816351 podStartE2EDuration="2.459816351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:20.457565012 +0000 UTC m=+1156.959475501" watchObservedRunningTime="2026-10-30 19:05:20.459816351 +0000 UTC m=+1156.961726837" Oct 30 19:05:21.356031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:21.355994 4789 generic.go:356] "Generic (PLEG): container finished" podID="0a528eea-eacd-49f4-8047-948171ab4b55" containerID="ffbd260bb4c6b631ff6750ac10fb6c86eac05a4ab7cb8dc2cda68297f1f6a7ba" exitCode=0 Oct 30 19:05:21.356454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:21.356027 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" event={"ID":"0a528eea-eacd-49f4-8047-948171ab4b55","Type":"ContainerDied","Data":"ffbd260bb4c6b631ff6750ac10fb6c86eac05a4ab7cb8dc2cda68297f1f6a7ba"} Oct 30 19:05:22.507613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.507588 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:22.555639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.555604 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\") pod \"0a528eea-eacd-49f4-8047-948171ab4b55\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " Oct 30 19:05:22.555824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.555806 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\" (UniqueName: \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\") pod \"0a528eea-eacd-49f4-8047-948171ab4b55\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " Oct 30 19:05:22.555915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.555850 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\") pod \"0a528eea-eacd-49f4-8047-948171ab4b55\" (UID: \"0a528eea-eacd-49f4-8047-948171ab4b55\") " Oct 30 19:05:22.556289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.556263 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca" pod "0a528eea-eacd-49f4-8047-948171ab4b55" (UID: "0a528eea-eacd-49f4-8047-948171ab4b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:22.557754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.557728 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca" pod "0a528eea-eacd-49f4-8047-948171ab4b55" (UID: "0a528eea-eacd-49f4-8047-948171ab4b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:22.557853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.557780 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm" pod "0a528eea-eacd-49f4-8047-948171ab4b55" (UID: "0a528eea-eacd-49f4-8047-948171ab4b55"). InnerVolumeSpecName "kube-api-access-2jkrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:22.657507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.657439 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkrm\" (UniqueName: \"kubernetes.io/projected/0a528eea-eacd-49f4-8047-948171ab4b55-kube-api-access-2jkrm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.657507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.657465 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.657507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:22.657475 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a528eea-eacd-49f4-8047-948171ab4b55-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:23.368696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:23.368662 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" Oct 30 19:05:23.368898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:23.368697 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7" event={"ID":"0a528eea-eacd-49f4-8047-948171ab4b55","Type":"ContainerDied","Data":"646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad"} Oct 30 19:05:23.368898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:23.368769 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="646223c29bdbafcdd8ce040f05ba9a2a73f5078b0b7721f59d6158ce1e3b13ad" Oct 30 19:05:26.370014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.369977 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:05:26.370446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.370392 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0a528eea-eacd-49f4-8047-948171ab4b55" containerName="revoke-keys" Oct 30 19:05:26.370446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.370404 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a528eea-eacd-49f4-8047-948171ab4b55" containerName="revoke-keys" Oct 30 19:05:26.370516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.370502 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a528eea-eacd-49f4-8047-948171ab4b55" containerName="revoke-keys" Oct 30 19:05:26.372678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.372657 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.422138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.422116 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-330025-openshift-default-dockercfg-9qb6q\"" Oct 30 19:05:26.471948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.471919 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:05:26.499778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.499744 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.499902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.499878 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.499969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.499952 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500009 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500114 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500168 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500275 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xzk\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500343 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.500424 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.500410 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601383 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xzk\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601421 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601448 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601492 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601571 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601628 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601680 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601709 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601752 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.601866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601836 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.602025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.601981 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.602061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.602037 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.602117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.602090 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.602173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.602156 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.604004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.603971 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.604128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.604046 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.659499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.659428 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.659629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.659581 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xzk\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:26.684584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:26.684548 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:27.112567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.112530 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:05:27.155345 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:27.155315 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d51dae9_9c7d_43a0_8618_b0e226c2f517.slice/crio-3cc8d94cb736011cddd71d0fea559f882a00f343f9fb0eb0a6f9cabedb32b095 WatchSource:0}: Error finding container 3cc8d94cb736011cddd71d0fea559f882a00f343f9fb0eb0a6f9cabedb32b095: Status 404 returned error can't find the container with id 3cc8d94cb736011cddd71d0fea559f882a00f343f9fb0eb0a6f9cabedb32b095 Oct 30 19:05:27.157526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.157492 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:27.157625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.157570 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:27.157625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.157612 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:27.392416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.392336 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" event={"ID":"7d51dae9-9c7d-43a0-8618-b0e226c2f517","Type":"ContainerStarted","Data":"30693127176a17c9146e99562f0f03f3e3adde6f66a9e68a82c2bad8e190f25a"} Oct 30 19:05:27.392416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.392389 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" event={"ID":"7d51dae9-9c7d-43a0-8618-b0e226c2f517","Type":"ContainerStarted","Data":"3cc8d94cb736011cddd71d0fea559f882a00f343f9fb0eb0a6f9cabedb32b095"} Oct 30 19:05:27.508859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.508803 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podStartSLOduration=1.508788466 podStartE2EDuration="1.508788466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:27.507513709 +0000 UTC m=+1164.009424195" watchObservedRunningTime="2026-10-30 19:05:27.508788466 +0000 UTC m=+1164.010698949" Oct 30 19:05:27.684856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:27.684833 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:28.686456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:28.686420 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:05:28.686837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:28.686519 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 19:05:29.685530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.685496 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:05:29.685679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.685547 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 19:05:29.713333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.713301 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd"] Oct 30 19:05:29.717444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.717414 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.766218 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.766184 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd"] Oct 30 19:05:29.785826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.785802 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:05:29.814232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.813468 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:05:29.814232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.813534 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:05:29.814232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.813562 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:29.839364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.839336 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92z4m\" (UniqueName: \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.839492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.839398 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.839549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.839499 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.940260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.940147 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92z4m\" (UniqueName: \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.940260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.940226 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.940460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.940302 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.940903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.940883 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:29.942545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:29.942527 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:30.015231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.014894 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92z4m\" (UniqueName: \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\") pod \"maas-api-revoke-sub-4691fea4-5wdvd\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:30.030799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.030769 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:30.417017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.416713 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd"] Oct 30 19:05:30.418381 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:30.418352 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400a8996_d6e5_4663_bdab_1273765d4eba.slice/crio-be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8 WatchSource:0}: Error finding container be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8: Status 404 returned error can't find the container with id be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8 Oct 30 19:05:30.429471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.429444 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" event={"ID":"400a8996-d6e5-4663-bdab-1273765d4eba","Type":"ContainerStarted","Data":"be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8"} Oct 30 19:05:30.685458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.685431 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:05:30.685593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:30.685485 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 19:05:31.448096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:31.448057 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" event={"ID":"400a8996-d6e5-4663-bdab-1273765d4eba","Type":"ContainerStarted","Data":"dbe2602f777de0e18f34394201751bd077068f37f1fb7e625383457c26bf1e00"} Oct 30 19:05:31.562830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:31.562771 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" podStartSLOduration=2.562758011 podStartE2EDuration="2.562758011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:31.560109653 +0000 UTC m=+1168.062020143" watchObservedRunningTime="2026-10-30 19:05:31.562758011 +0000 UTC m=+1168.064668493" Oct 30 19:05:31.685038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:31.685004 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:05:31.685264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:31.685061 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 19:05:32.456597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:32.456557 4789 generic.go:356] "Generic (PLEG): container finished" podID="400a8996-d6e5-4663-bdab-1273765d4eba" containerID="dbe2602f777de0e18f34394201751bd077068f37f1fb7e625383457c26bf1e00" exitCode=0 Oct 30 19:05:32.456957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:32.456624 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" event={"ID":"400a8996-d6e5-4663-bdab-1273765d4eba","Type":"ContainerDied","Data":"dbe2602f777de0e18f34394201751bd077068f37f1fb7e625383457c26bf1e00"} Oct 30 19:05:32.685516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:32.685484 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:05:32.685680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:32.685536 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 19:05:33.624935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.624902 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:33.676607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.676572 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\" (UniqueName: \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\") pod \"400a8996-d6e5-4663-bdab-1273765d4eba\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " Oct 30 19:05:33.676774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.676631 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\") pod \"400a8996-d6e5-4663-bdab-1273765d4eba\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " Oct 30 19:05:33.676774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.676696 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\") pod \"400a8996-d6e5-4663-bdab-1273765d4eba\" (UID: \"400a8996-d6e5-4663-bdab-1273765d4eba\") " Oct 30 19:05:33.677143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.677109 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca" pod "400a8996-d6e5-4663-bdab-1273765d4eba" (UID: "400a8996-d6e5-4663-bdab-1273765d4eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:33.678734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.678708 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m" pod "400a8996-d6e5-4663-bdab-1273765d4eba" (UID: "400a8996-d6e5-4663-bdab-1273765d4eba"). InnerVolumeSpecName "kube-api-access-92z4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:33.678734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.678717 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca" pod "400a8996-d6e5-4663-bdab-1273765d4eba" (UID: "400a8996-d6e5-4663-bdab-1273765d4eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:33.749017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.748933 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:33.749929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.749905 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:33.750644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.750626 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:05:33.778111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.778082 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92z4m\" (UniqueName: \"kubernetes.io/projected/400a8996-d6e5-4663-bdab-1273765d4eba-kube-api-access-92z4m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:33.778111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.778108 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:33.778304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:33.778119 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a8996-d6e5-4663-bdab-1273765d4eba-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:34.469247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:34.469190 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" Oct 30 19:05:34.469247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:34.469222 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd" event={"ID":"400a8996-d6e5-4663-bdab-1273765d4eba","Type":"ContainerDied","Data":"be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8"} Oct 30 19:05:34.469468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:34.469277 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be2d39c73e7fdbf1cf38f974e0a81a85f7bd0385894c8e44d64760d119c76dc8" Oct 30 19:05:41.518602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.518566 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z"] Oct 30 19:05:41.523524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.523486 4789 cpu_manager.go:414] "containerMap: removing container" podUID="400a8996-d6e5-4663-bdab-1273765d4eba" containerName="revoke-keys" Oct 30 19:05:41.523524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.523521 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a8996-d6e5-4663-bdab-1273765d4eba" containerName="revoke-keys" Oct 30 19:05:41.523796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.523782 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="400a8996-d6e5-4663-bdab-1273765d4eba" containerName="revoke-keys" Oct 30 19:05:41.527225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.527183 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.577811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.577778 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z"] Oct 30 19:05:41.620690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.620664 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:05:41.620690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.620668 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:41.620867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.620702 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:05:41.622665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.622645 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:05:41.659225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.659177 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmwk\" (UniqueName: \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.659396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.659320 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.659461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.659400 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.760260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.760188 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.760451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.760373 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.760565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.760545 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmwk\" (UniqueName: \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.761119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.761093 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.763340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.763312 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.823202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.823128 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmwk\" (UniqueName: \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\") pod \"maas-api-revoke-sub-d87ebfad-bgq8z\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:41.845263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:41.845229 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:42.226036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:42.226000 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z"] Oct 30 19:05:42.234169 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:42.234126 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ff7041_ecf0_45d1_ab2b_38c35aeac3c1.slice/crio-b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5 WatchSource:0}: Error finding container b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5: Status 404 returned error can't find the container with id b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5 Oct 30 19:05:42.520463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:42.520427 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" event={"ID":"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1","Type":"ContainerStarted","Data":"36053cc2722b6d0a72018386738240bec4c3af5cdfd0dab2de11b7bd3b50992f"} Oct 30 19:05:42.520926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:42.520496 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" event={"ID":"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1","Type":"ContainerStarted","Data":"b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5"} Oct 30 19:05:42.631984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:42.631922 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" podStartSLOduration=1.631903816 podStartE2EDuration="1.631903816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:42.630771194 +0000 UTC m=+1179.132681681" watchObservedRunningTime="2026-10-30 19:05:42.631903816 +0000 UTC m=+1179.133814303" Oct 30 19:05:43.527540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:43.527507 4789 generic.go:356] "Generic (PLEG): container finished" podID="c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" containerID="36053cc2722b6d0a72018386738240bec4c3af5cdfd0dab2de11b7bd3b50992f" exitCode=0 Oct 30 19:05:43.527950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:43.527601 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" event={"ID":"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1","Type":"ContainerDied","Data":"36053cc2722b6d0a72018386738240bec4c3af5cdfd0dab2de11b7bd3b50992f"} Oct 30 19:05:44.534776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.534742 4789 generic.go:356] "Generic (PLEG): container finished" podID="e3b25566-df8b-4147-8bd3-1382cf94587f" containerID="6c8f2958d903c378886a66ace22db80ba2a024d63c2a491950722e914787b957" exitCode=0 Oct 30 19:05:44.535144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.534819 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" event={"ID":"e3b25566-df8b-4147-8bd3-1382cf94587f","Type":"ContainerDied","Data":"6c8f2958d903c378886a66ace22db80ba2a024d63c2a491950722e914787b957"} Oct 30 19:05:44.691400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.691371 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:44.789433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.789371 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\") pod \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " Oct 30 19:05:44.789575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.789466 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\" (UniqueName: \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\") pod \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " Oct 30 19:05:44.789575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.789560 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\") pod \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\" (UID: \"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1\") " Oct 30 19:05:44.789762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.789741 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca" pod "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" (UID: "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:44.790027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.789991 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:44.791621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.791594 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk" pod "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" (UID: "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1"). InnerVolumeSpecName "kube-api-access-prmwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:44.791708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.791661 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca" pod "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" (UID: "c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:44.890726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.890696 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:44.890726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:44.890724 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prmwk\" (UniqueName: \"kubernetes.io/projected/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1-kube-api-access-prmwk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:45.541256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.541222 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" Oct 30 19:05:45.541256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.541237 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z" event={"ID":"c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1","Type":"ContainerDied","Data":"b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5"} Oct 30 19:05:45.541712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.541309 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b0ca417a627caf6f759b18733616c093076812f7e5807a3d6520a5d833a9d5" Oct 30 19:05:45.685185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.685159 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:05:45.798651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.798581 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\") pod \"e3b25566-df8b-4147-8bd3-1382cf94587f\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " Oct 30 19:05:45.798797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.798660 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\" (UniqueName: \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\") pod \"e3b25566-df8b-4147-8bd3-1382cf94587f\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " Oct 30 19:05:45.798797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.798736 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\") pod \"e3b25566-df8b-4147-8bd3-1382cf94587f\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " Oct 30 19:05:45.798797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.798792 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\") pod \"e3b25566-df8b-4147-8bd3-1382cf94587f\" (UID: \"e3b25566-df8b-4147-8bd3-1382cf94587f\") " Oct 30 19:05:45.799028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.798933 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle" pod "e3b25566-df8b-4147-8bd3-1382cf94587f" (UID: "e3b25566-df8b-4147-8bd3-1382cf94587f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:45.799163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.799145 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle" pod "e3b25566-df8b-4147-8bd3-1382cf94587f" (UID: "e3b25566-df8b-4147-8bd3-1382cf94587f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:45.799323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.799292 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path" pod "e3b25566-df8b-4147-8bd3-1382cf94587f" (UID: "e3b25566-df8b-4147-8bd3-1382cf94587f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:45.799588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.799552 4789 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e3b25566-df8b-4147-8bd3-1382cf94587f-archives-path\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:45.799776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.799760 4789 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-trusted-ca-bundle\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:45.799852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.799782 4789 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3b25566-df8b-4147-8bd3-1382cf94587f-service-ca-bundle\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:45.801678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.801656 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs" pod "e3b25566-df8b-4147-8bd3-1382cf94587f" (UID: "e3b25566-df8b-4147-8bd3-1382cf94587f"). InnerVolumeSpecName "kube-api-access-fb6bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:45.901073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:45.901042 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6bs\" (UniqueName: \"kubernetes.io/projected/e3b25566-df8b-4147-8bd3-1382cf94587f-kube-api-access-fb6bs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:46.548472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:46.548438 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" Oct 30 19:05:46.548947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:46.548471 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rzr4l-wkkhj" event={"ID":"e3b25566-df8b-4147-8bd3-1382cf94587f","Type":"ContainerDied","Data":"1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff"} Oct 30 19:05:46.548947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:46.548515 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e88f532fab811876aef8d52feee3375edb7a218dbbcca66e6edbba7e63394ff" Oct 30 19:05:48.197073 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:05:48.197034 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:05:58.771786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.771739 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:05:58.772513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772418 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e3b25566-df8b-4147-8bd3-1382cf94587f" containerName="insights-gathering" Oct 30 19:05:58.772513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772439 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b25566-df8b-4147-8bd3-1382cf94587f" containerName="insights-gathering" Oct 30 19:05:58.772513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772486 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" containerName="revoke-keys" Oct 30 19:05:58.772513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772495 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" containerName="revoke-keys" Oct 30 19:05:58.772718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772604 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b25566-df8b-4147-8bd3-1382cf94587f" containerName="insights-gathering" Oct 30 19:05:58.772718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.772623 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" containerName="revoke-keys" Oct 30 19:05:58.776247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.776226 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.825620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.825600 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2301cb-openshift-default-dockercfg-r29tl\"" Oct 30 19:05:58.874806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.874784 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:05:58.926656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926628 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.926778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926671 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.926778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926691 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnrk\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.926928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926816 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.926985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926925 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.926985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926975 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.927074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.926995 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.927074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.927033 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:58.927074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:58.927055 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.027590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.027503 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.027590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.027551 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.027590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.027590 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.027862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.027776 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnrk\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.027993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.027972 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028031 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028170 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028257 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028291 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028313 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028342 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028663 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028800 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.028998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.028968 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.030410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.030388 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.030525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.030501 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.085534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.085504 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.085684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.085511 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnrk\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.089040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.089017 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:05:59.528983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.528941 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:05:59.534674 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:05:59.534642 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467bc477_7e67_4846_b5a0_956fff079645.slice/crio-0c74b79b86b8fdb954d4c9936029264b8797249e462852f431916ae3821e2093 WatchSource:0}: Error finding container 0c74b79b86b8fdb954d4c9936029264b8797249e462852f431916ae3821e2093: Status 404 returned error can't find the container with id 0c74b79b86b8fdb954d4c9936029264b8797249e462852f431916ae3821e2093 Oct 30 19:05:59.537618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.537588 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:59.537715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.537699 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:59.537777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.537748 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:05:59.633195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.633160 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" event={"ID":"467bc477-7e67-4846-b5a0-956fff079645","Type":"ContainerStarted","Data":"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66"} Oct 30 19:05:59.633371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.633268 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" event={"ID":"467bc477-7e67-4846-b5a0-956fff079645","Type":"ContainerStarted","Data":"0c74b79b86b8fdb954d4c9936029264b8797249e462852f431916ae3821e2093"} Oct 30 19:05:59.766332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:05:59.766282 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podStartSLOduration=1.7662693699999998 podStartE2EDuration="1.76626937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:59.764587455 +0000 UTC m=+1196.266497943" watchObservedRunningTime="2026-10-30 19:05:59.76626937 +0000 UTC m=+1196.268179856" Oct 30 19:06:00.076036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.075997 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:06:00.080086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.080063 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.089331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.089313 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:06:00.090471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.090451 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:00.090558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.090488 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:00.243635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.243594 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.243887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.243860 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244013 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244046 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244104 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgnp\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244134 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244281 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244333 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.244430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.244404 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.287280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.287253 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:06:00.345864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.345771 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.346618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.346531 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.347086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.346576 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.347276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.347253 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.347740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.347724 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.348256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.348127 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.348615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.348597 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgnp\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.348761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.348739 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.348944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.348929 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.349587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.348775 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.349805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.349750 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.350492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.350366 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.350632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.350615 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.351186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.351071 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.352455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.352413 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.353197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.353179 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.436983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.436915 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgnp\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.437698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.437668 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-6g9z6\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:00.705233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:00.704382 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:01.089393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.089318 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:01.089393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.089365 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:01.144443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.144403 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:06:01.149865 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:06:01.149838 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83da78a7_ce06_4c04_8801_aa26491c3e26.slice/crio-c6145d98d3b748c9b326f7c9fe864b55a098f6fbcfe993fa71ac040466eb60e2 WatchSource:0}: Error finding container c6145d98d3b748c9b326f7c9fe864b55a098f6fbcfe993fa71ac040466eb60e2: Status 404 returned error can't find the container with id c6145d98d3b748c9b326f7c9fe864b55a098f6fbcfe993fa71ac040466eb60e2 Oct 30 19:06:01.152488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.152458 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:01.152576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.152540 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:01.152613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.152573 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:01.649473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.649417 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" event={"ID":"83da78a7-ce06-4c04-8801-aa26491c3e26","Type":"ContainerStarted","Data":"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f"} Oct 30 19:06:01.649674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.649491 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" event={"ID":"83da78a7-ce06-4c04-8801-aa26491c3e26","Type":"ContainerStarted","Data":"c6145d98d3b748c9b326f7c9fe864b55a098f6fbcfe993fa71ac040466eb60e2"} Oct 30 19:06:01.718233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.714303 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:01.782866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:01.782796 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podStartSLOduration=1.7827771559999999 podStartE2EDuration="1.782777156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:01.776796144 +0000 UTC m=+1198.278706630" watchObservedRunningTime="2026-10-30 19:06:01.782777156 +0000 UTC m=+1198.284687644" Oct 30 19:06:02.089822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:02.089788 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:02.090298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:02.090270 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:02.709024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:02.708986 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:02.709345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:02.709314 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:03.090095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.090008 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:03.090095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.090081 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:03.738229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.731494 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:03.738229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.731553 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:03.860604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.860570 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:06:03.872457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:03.872427 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:06:04.090045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:04.089971 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:04.090045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:04.090016 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:04.707233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:04.706561 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:04.707233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:04.706633 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:05.090434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:05.090347 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:06:05.090813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:05.090420 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 19:06:05.709558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:05.709521 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:05.709884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:05.709853 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:06.477141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:06.477105 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:06:06.477901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:06.477876 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:06:06.480205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:06.480187 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:06:06.709574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:06.709536 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:06.709820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:06.709790 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:07.706422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:07.706386 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:07.706839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:07.706447 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:08.705726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:08.705691 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:06:08.705901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:08.705749 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 19:06:09.957629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:09.957593 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:09.959018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:09.958979 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:09.959739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:09.959715 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:06:15.202550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.202476 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:06:15.207381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.207358 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284365 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284404 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284504 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284553 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6p8x\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284657 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284681 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284783 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284824 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.284878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.284870 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386023 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386061 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6p8x\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386093 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386116 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386150 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386171 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386190 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386303 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386333 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386618 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386651 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386737 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.386802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.386771 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.387076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.387050 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.388626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.388606 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.388778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.388761 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.407120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.407093 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:06:15.444434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.444401 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.444564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.444482 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6p8x\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-54987\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.519977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.519900 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:15.950442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.950401 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:06:15.956177 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:06:15.956150 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec37f1c_c7d6_4e7f_bed0_7ace8b136e7c.slice/crio-201fefe78efc9007ba1c168bfd4679599427253568b9e935b268e62b960ee3bb WatchSource:0}: Error finding container 201fefe78efc9007ba1c168bfd4679599427253568b9e935b268e62b960ee3bb: Status 404 returned error can't find the container with id 201fefe78efc9007ba1c168bfd4679599427253568b9e935b268e62b960ee3bb Oct 30 19:06:15.958642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.958611 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:15.958720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.958678 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:15.958720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:15.958704 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:06:16.780723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:16.780682 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" event={"ID":"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c","Type":"ContainerStarted","Data":"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623"} Oct 30 19:06:16.781149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:16.780744 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" event={"ID":"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c","Type":"ContainerStarted","Data":"201fefe78efc9007ba1c168bfd4679599427253568b9e935b268e62b960ee3bb"} Oct 30 19:06:17.520569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:17.520532 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:17.522530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:17.522502 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:17.522663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:17.522553 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:18.220187 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:06:18.220145 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:06:18.520758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:18.520662 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:18.520758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:18.520732 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:19.520805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:19.520759 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:19.521364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:19.520840 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:20.520694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:20.520653 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:20.520900 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:20.520727 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:21.522637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:21.522598 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:21.523179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:21.522676 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:22.521084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:22.521033 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 19:06:22.521317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:22.521117 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 19:06:24.336003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:24.335967 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:24.336538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:24.336347 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:24.337260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:24.337234 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:06:24.498306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:24.498253 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podStartSLOduration=9.498239021 podStartE2EDuration="9.498239021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:16.907562468 +0000 UTC m=+1213.409472955" watchObservedRunningTime="2026-10-30 19:06:24.498239021 +0000 UTC m=+1221.000149562" Oct 30 19:06:38.963754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:38.963716 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 19:06:38.969062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:38.969038 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.024143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.024099 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 19:06:39.062679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.062631 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.062854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.062751 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.062951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.062847 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.062951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.062924 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.063071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.062967 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jxc\" (UniqueName: \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.063071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.063022 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.066305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.066281 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1323a7\"/\"distinct-w4-1323a7-kserve-self-signed-certs\"" Oct 30 19:06:39.164018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.163977 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jxc\" (UniqueName: \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164040 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164102 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164166 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164268 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164317 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164619 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164670 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.164753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.164721 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.166437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.166415 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.166697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.166679 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.223076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.222994 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jxc\" (UniqueName: \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\") pod \"distinct-w4-1323a7-kserve-6fb6766dfd-xgs72\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.281508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.281469 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:39.724154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.724109 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 19:06:39.730919 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:06:39.730891 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a27d90e_3c13_438b_a204_74dc08b39517.slice/crio-6d8f19751efe5b12ca4793714b7ea8dd214c6bf085bbe211b7747a595c1ab21f WatchSource:0}: Error finding container 6d8f19751efe5b12ca4793714b7ea8dd214c6bf085bbe211b7747a595c1ab21f: Status 404 returned error can't find the container with id 6d8f19751efe5b12ca4793714b7ea8dd214c6bf085bbe211b7747a595c1ab21f Oct 30 19:06:39.952927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.952892 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" event={"ID":"3a27d90e-3c13-438b-a204-74dc08b39517","Type":"ContainerStarted","Data":"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d"} Oct 30 19:06:39.952927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:39.952932 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" event={"ID":"3a27d90e-3c13-438b-a204-74dc08b39517","Type":"ContainerStarted","Data":"6d8f19751efe5b12ca4793714b7ea8dd214c6bf085bbe211b7747a595c1ab21f"} Oct 30 19:06:40.066763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:40.066659 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" podStartSLOduration=2.0666458 podStartE2EDuration="2.0666458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:40.064195711 +0000 UTC m=+1236.566106198" watchObservedRunningTime="2026-10-30 19:06:40.0666458 +0000 UTC m=+1236.568556285" Oct 30 19:06:40.282395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:40.282366 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:48.196805 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:06:48.196766 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:06:49.282460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:49.282417 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:49.299327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:49.299294 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:06:50.049542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:06:50.049510 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 19:07:03.867281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:03.867248 4789 scope.go:122] "RemoveContainer" containerID="d7f02e8a7492e88ffa8d61ebbed1b0ee5a36887fe555a05d2dd03c619acad60d" Oct 30 19:07:03.879003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:03.878973 4789 scope.go:122] "RemoveContainer" containerID="9927e04d0e737f6ba3959431607e97b410ff3ed3f0f8248557ce3de794ea429d" Oct 30 19:07:03.890369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:03.890345 4789 scope.go:122] "RemoveContainer" containerID="109dec1fad3e3524313d10c25fb9361708273803e6dec87ee34fa9c620d0ee5d" Oct 30 19:07:03.899133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:03.899107 4789 scope.go:122] "RemoveContainer" containerID="4abd74d2b8645b0beef7901e13057d7697f17898f2adc9066805d86c5cf6382a" Oct 30 19:07:03.910373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:03.910353 4789 scope.go:122] "RemoveContainer" containerID="de3d19b143b9eb6afbace88413c32721a1a027227ed01d9a5dc836bb5d495687" Oct 30 19:07:18.196477 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:07:18.196435 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:07:26.003234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.002334 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:07:26.003234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.002423 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:07:26.185165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.185122 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:07:26.214880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.214843 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.250242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.250200 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:07:26.283955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.283853 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-22fb18\"/\"distinct-w3-22fb18-kserve-self-signed-certs\"" Oct 30 19:07:26.330851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.330818 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.331112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.330866 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.331112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.330916 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.331112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.330940 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8rn\" (UniqueName: \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.331112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.331070 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.331376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.331148 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.432563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.432529 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.432780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.432599 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.432780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.432648 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.432983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.432961 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.433226 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.433182 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8rn\" (UniqueName: \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.433557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.433111 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.433646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.433060 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.433646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.433356 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.433780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.433754 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.436989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.436961 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.437255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.437231 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.492725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.492688 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8rn\" (UniqueName: \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\") pod \"distinct-w3-22fb18-kserve-8654f78695-g9ftq\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.555275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.555160 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:26.966586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:26.966547 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:07:26.973149 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:07:26.973121 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e67e16_4fe3_4732_9da8_da86d4f1af92.slice/crio-361ef8c06199232a1381fb6b4347a28508c3afa7aef143357c9fbca3085dceff WatchSource:0}: Error finding container 361ef8c06199232a1381fb6b4347a28508c3afa7aef143357c9fbca3085dceff: Status 404 returned error can't find the container with id 361ef8c06199232a1381fb6b4347a28508c3afa7aef143357c9fbca3085dceff Oct 30 19:07:27.281259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:27.281131 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" event={"ID":"58e67e16-4fe3-4732-9da8-da86d4f1af92","Type":"ContainerStarted","Data":"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c"} Oct 30 19:07:27.281259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:27.281175 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" event={"ID":"58e67e16-4fe3-4732-9da8-da86d4f1af92","Type":"ContainerStarted","Data":"361ef8c06199232a1381fb6b4347a28508c3afa7aef143357c9fbca3085dceff"} Oct 30 19:07:27.393281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:27.393229 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podStartSLOduration=1.393200254 podStartE2EDuration="1.393200254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:27.390926027 +0000 UTC m=+1283.892836513" watchObservedRunningTime="2026-10-30 19:07:27.393200254 +0000 UTC m=+1283.895110739" Oct 30 19:07:27.555362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:27.555279 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:36.556047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:36.556009 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:36.568393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:36.568369 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:37.392373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:37.392342 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:07:48.196378 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:07:48.196297 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:07:59.819448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.819406 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:07:59.823776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.823750 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.874418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.874391 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-22fb18-openshift-default-dockercfg-pqpjp\"" Oct 30 19:07:59.923596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.923560 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:07:59.945022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.944991 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945048 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945186 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945293 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945365 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945464 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945521 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945553 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:07:59.945707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:07:59.945668 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hx9d\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047784 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047831 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047857 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hx9d\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047896 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047925 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047947 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.047996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.047968 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048023 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048061 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048282 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048339 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048444 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048508 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.048654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.048634 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.050279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.050258 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.050546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.050526 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.110593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.110524 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hx9d\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.140043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.140010 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.437877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.437845 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:00.899028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.898948 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:08:00.912256 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:08:00.912200 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afc72ed_573a_4879_9388_4c6d208c953e.slice/crio-61df095c45a198706ee67b7f3a20efbac1dde601408264d30614b05dacd7ca2a WatchSource:0}: Error finding container 61df095c45a198706ee67b7f3a20efbac1dde601408264d30614b05dacd7ca2a: Status 404 returned error can't find the container with id 61df095c45a198706ee67b7f3a20efbac1dde601408264d30614b05dacd7ca2a Oct 30 19:08:00.914134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.914100 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:08:00.914342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.914175 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:08:00.914342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:00.914203 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:08:01.524315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:01.524261 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" event={"ID":"0afc72ed-573a-4879-9388-4c6d208c953e","Type":"ContainerStarted","Data":"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f"} Oct 30 19:08:01.524531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:01.524340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" event={"ID":"0afc72ed-573a-4879-9388-4c6d208c953e","Type":"ContainerStarted","Data":"61df095c45a198706ee67b7f3a20efbac1dde601408264d30614b05dacd7ca2a"} Oct 30 19:08:01.734139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:01.734087 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podStartSLOduration=2.7340728309999998 podStartE2EDuration="2.734072831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:01.686545948 +0000 UTC m=+1318.188456434" watchObservedRunningTime="2026-10-30 19:08:01.734072831 +0000 UTC m=+1318.235983386" Oct 30 19:08:01.952035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:01.952000 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x"] Oct 30 19:08:01.956040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:01.956024 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.008233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.007568 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:08:02.008233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.007777 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:08:02.008233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.007845 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:02.008519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.008347 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:08:02.023231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.022378 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x"] Oct 30 19:08:02.069937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.069901 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vz8w\" (UniqueName: \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.069937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.069942 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.070133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.069969 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.170977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.170944 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vz8w\" (UniqueName: \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.171149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.170981 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.171149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.171010 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.171661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.171625 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.173479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.173455 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.230039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.229957 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vz8w\" (UniqueName: \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\") pod \"maas-api-revoke-sub-0711725c-9v75x\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.271752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.271722 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:02.442670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.442322 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:02.442670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.442419 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:02.442670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.442464 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:02.682501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:02.682418 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x"] Oct 30 19:08:02.688109 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:08:02.688077 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a WatchSource:0}: Error finding container 7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a: Status 404 returned error can't find the container with id 7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a Oct 30 19:08:03.438706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:03.438637 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:03.438706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:03.438691 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:03.539134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:03.539091 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" event={"ID":"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04","Type":"ContainerStarted","Data":"aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e"} Oct 30 19:08:03.539348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:03.539163 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" event={"ID":"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04","Type":"ContainerStarted","Data":"7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a"} Oct 30 19:08:03.650967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:03.650917 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" podStartSLOduration=2.650903078 podStartE2EDuration="2.650903078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:03.648756715 +0000 UTC m=+1320.150667203" watchObservedRunningTime="2026-10-30 19:08:03.650903078 +0000 UTC m=+1320.152813563" Oct 30 19:08:04.438485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:04.438450 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:04.438683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:04.438507 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:04.546639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:04.546603 4789 generic.go:356] "Generic (PLEG): container finished" podID="6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" containerID="aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e" exitCode=0 Oct 30 19:08:04.547039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:04.546691 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" event={"ID":"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04","Type":"ContainerDied","Data":"aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e"} Oct 30 19:08:05.438952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.438915 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:05.439155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.438968 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:05.733875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.733850 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:05.831501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.831469 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\" (UniqueName: \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\") pod \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " Oct 30 19:08:05.831680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.831556 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\") pod \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " Oct 30 19:08:05.831680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.831581 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\") pod \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\" (UID: \"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04\") " Oct 30 19:08:05.831915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.831887 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca" pod "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" (UID: "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:05.832063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.832049 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:05.833572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.833549 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca" pod "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" (UID: "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:05.833671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.833609 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w" pod "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" (UID: "6b556f1e-cdde-41e2-8ff5-6c5e6abacb04"). InnerVolumeSpecName "kube-api-access-5vz8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:05.933181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.933146 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vz8w\" (UniqueName: \"kubernetes.io/projected/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-kube-api-access-5vz8w\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:05.933181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:05.933176 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:06.438230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:06.438175 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:06.438406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:06.438267 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:06.565367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:06.562722 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" event={"ID":"6b556f1e-cdde-41e2-8ff5-6c5e6abacb04","Type":"ContainerDied","Data":"7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a"} Oct 30 19:08:06.565367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:06.562819 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9c5f32e2a42d793d98eafe88e1ab21ceaf02c78dd5c61ad750db0194bfb32a" Oct 30 19:08:06.565367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:06.562913 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x" Oct 30 19:08:07.438481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:07.438447 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:08:07.438883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:07.438512 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 19:08:08.444631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:08.444600 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:08.445126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:08.444999 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:08.445656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:08.445634 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:08:08.868579 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:08.868498 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:11.810439 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:11.810398 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:15.842161 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.842127 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4"] Oct 30 19:08:15.842891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.842870 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" containerName="revoke-keys" Oct 30 19:08:15.842949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.842895 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" containerName="revoke-keys" Oct 30 19:08:15.843041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.843028 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" containerName="revoke-keys" Oct 30 19:08:15.847080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.847059 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:15.882676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.882644 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4"] Oct 30 19:08:15.896247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.896196 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:15.924929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.924900 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:08:15.924929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.924900 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:08:15.925116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.924904 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:08:15.941394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.941368 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplsk\" (UniqueName: \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:15.941490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.941405 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:15.941490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:15.941430 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.042642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.042604 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplsk\" (UniqueName: \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.042821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.042656 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.042821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.042683 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.043326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.043307 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.045197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.045177 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.100954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.100894 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplsk\" (UniqueName: \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\") pod \"maas-api-revoke-sub-ff3ffb00-l67f4\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.160259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.160227 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:16.566961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.566925 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4"] Oct 30 19:08:16.635449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:16.635400 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" event={"ID":"b1586106-aa3e-4956-81b0-42923f9014b5","Type":"ContainerStarted","Data":"92c3ba00f6f9ec519b37da15f71f98091635ab7496eaea893235de3cdff22acd"} Oct 30 19:08:17.644795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:17.644756 4789 generic.go:356] "Generic (PLEG): container finished" podID="b1586106-aa3e-4956-81b0-42923f9014b5" containerID="22debd74ca1aea10645d9f81620ad5e21c1d11aaa5782eb296cac2f8c877a777" exitCode=0 Oct 30 19:08:17.645228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:17.644834 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" event={"ID":"b1586106-aa3e-4956-81b0-42923f9014b5","Type":"ContainerDied","Data":"22debd74ca1aea10645d9f81620ad5e21c1d11aaa5782eb296cac2f8c877a777"} Oct 30 19:08:18.195392 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:18.195356 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:08:18.816994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.816969 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:18.872849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.872822 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\") pod \"b1586106-aa3e-4956-81b0-42923f9014b5\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " Oct 30 19:08:18.872957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.872871 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\") pod \"b1586106-aa3e-4956-81b0-42923f9014b5\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " Oct 30 19:08:18.872957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.872934 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\" (UniqueName: \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\") pod \"b1586106-aa3e-4956-81b0-42923f9014b5\" (UID: \"b1586106-aa3e-4956-81b0-42923f9014b5\") " Oct 30 19:08:18.873305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.873283 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca" pod "b1586106-aa3e-4956-81b0-42923f9014b5" (UID: "b1586106-aa3e-4956-81b0-42923f9014b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:18.874899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.874873 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk" pod "b1586106-aa3e-4956-81b0-42923f9014b5" (UID: "b1586106-aa3e-4956-81b0-42923f9014b5"). InnerVolumeSpecName "kube-api-access-dplsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:18.874899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.874886 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca" pod "b1586106-aa3e-4956-81b0-42923f9014b5" (UID: "b1586106-aa3e-4956-81b0-42923f9014b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:18.934481 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:18.934455 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:18.973927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.973898 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplsk\" (UniqueName: \"kubernetes.io/projected/b1586106-aa3e-4956-81b0-42923f9014b5-kube-api-access-dplsk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:18.974039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.973928 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:18.974039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:18.973943 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1586106-aa3e-4956-81b0-42923f9014b5-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:19.658931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:19.658899 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" Oct 30 19:08:19.659105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:19.658926 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4" event={"ID":"b1586106-aa3e-4956-81b0-42923f9014b5","Type":"ContainerDied","Data":"92c3ba00f6f9ec519b37da15f71f98091635ab7496eaea893235de3cdff22acd"} Oct 30 19:08:19.659105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:08:19.658979 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c3ba00f6f9ec519b37da15f71f98091635ab7496eaea893235de3cdff22acd" Oct 30 19:08:26.867837 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:26.867801 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:28.748233 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:28.725920 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:28.750432 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:28.733907 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:28.958408 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:28.958368 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:39.022246 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:39.022192 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:41.812646 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:41.812606 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:48.195715 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:48.195676 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:08:49.087555 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:49.087519 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:56.867886 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:56.867849 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:08:59.101543 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:08:59.101499 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b556f1e_cdde_41e2_8ff5_6c5e6abacb04.slice/crio-aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:09:18.199038 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:09:18.198945 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:09:22.060900 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:22.060860 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr"] Oct 30 19:09:22.108840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:22.108800 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-450333b6-xvhxr"] Oct 30 19:09:23.667315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:23.667273 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ca318c-08ec-4967-9574-c287bea4f15a" path="/var/lib/kubelet/pods/49ca318c-08ec-4967-9574-c287bea4f15a/volumes" Oct 30 19:09:25.124005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.123970 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7"] Oct 30 19:09:25.124522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.124505 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b1586106-aa3e-4956-81b0-42923f9014b5" containerName="revoke-keys" Oct 30 19:09:25.124522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.124523 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1586106-aa3e-4956-81b0-42923f9014b5" containerName="revoke-keys" Oct 30 19:09:25.124644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.124612 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1586106-aa3e-4956-81b0-42923f9014b5" containerName="revoke-keys" Oct 30 19:09:25.128012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.127985 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.177300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.177273 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:09:25.223400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.223370 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:25.223400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.223375 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:09:25.223576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.223535 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:09:25.225103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.225084 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.225167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.225125 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.225167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.225148 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmt99\" (UniqueName: \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.226288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.226268 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7"] Oct 30 19:09:25.326079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.326052 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.326250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.326096 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.326250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.326118 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmt99\" (UniqueName: \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.326778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.326751 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.328319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.328302 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.383983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.383932 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmt99\" (UniqueName: \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\") pod \"maas-api-revoke-sub-b03ad442-szrj7\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.441831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.441809 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:25.824034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:25.824001 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7"] Oct 30 19:09:25.831501 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:09:25.831456 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b601f1b_007d_4f2b_aadd_9a48fb3d1ea2.slice/crio-672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114 WatchSource:0}: Error finding container 672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114: Status 404 returned error can't find the container with id 672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114 Oct 30 19:09:26.156291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:26.156259 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" event={"ID":"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2","Type":"ContainerStarted","Data":"9aae8eb9651f80c1737e2ab626e0a935c67cde561b98eddb950c59b5080596b6"} Oct 30 19:09:26.156641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:26.156312 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" event={"ID":"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2","Type":"ContainerStarted","Data":"672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114"} Oct 30 19:09:26.265942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:26.265885 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" podStartSLOduration=1.265867299 podStartE2EDuration="1.265867299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:26.265567748 +0000 UTC m=+1402.767478237" watchObservedRunningTime="2026-10-30 19:09:26.265867299 +0000 UTC m=+1402.767777785" Oct 30 19:09:27.163095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:27.163062 4789 generic.go:356] "Generic (PLEG): container finished" podID="0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" containerID="9aae8eb9651f80c1737e2ab626e0a935c67cde561b98eddb950c59b5080596b6" exitCode=0 Oct 30 19:09:27.163513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:27.163148 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" event={"ID":"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2","Type":"ContainerDied","Data":"9aae8eb9651f80c1737e2ab626e0a935c67cde561b98eddb950c59b5080596b6"} Oct 30 19:09:28.310842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.310811 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:28.357961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.357932 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\") pod \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " Oct 30 19:09:28.358082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.357996 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\" (UniqueName: \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\") pod \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " Oct 30 19:09:28.358082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.358033 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\") pod \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\" (UID: \"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2\") " Oct 30 19:09:28.358465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.358433 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca" pod "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" (UID: "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:28.359950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.359922 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca" pod "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" (UID: "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:28.360047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.359991 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99" pod "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" (UID: "0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2"). InnerVolumeSpecName "kube-api-access-bmt99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:28.459988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.459953 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmt99\" (UniqueName: \"kubernetes.io/projected/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-kube-api-access-bmt99\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:28.460140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.459992 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:28.460140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:28.460008 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:29.175624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:29.175594 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" Oct 30 19:09:29.175624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:29.175601 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7" event={"ID":"0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2","Type":"ContainerDied","Data":"672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114"} Oct 30 19:09:29.175825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:29.175650 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="672d5b36f65b285c00194c40cda7da9306f2c2a5d4041aa41549c36c179e6114" Oct 30 19:09:40.049190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:40.049153 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql"] Oct 30 19:09:40.095184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:40.095153 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e493f7c-d6cql"] Oct 30 19:09:41.667969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:41.667939 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1263749-9956-4e10-8dab-914a832e2040" path="/var/lib/kubelet/pods/e1263749-9956-4e10-8dab-914a832e2040/volumes" Oct 30 19:09:48.196579 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:09:48.196534 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:09:53.060710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:53.060676 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2"] Oct 30 19:09:53.107006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:53.106975 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1b57173-qdtj2"] Oct 30 19:09:53.668302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:53.668262 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8d584b-bb01-4f4c-9da0-b60d115bdbc1" path="/var/lib/kubelet/pods/bb8d584b-bb01-4f4c-9da0-b60d115bdbc1/volumes" Oct 30 19:09:59.050598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:59.050553 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk"] Oct 30 19:09:59.102227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:59.097403 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4884b00b-k8pfk"] Oct 30 19:09:59.668747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:09:59.668716 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da0c03e-6ec9-47c6-aa85-3d49864d1a9b" path="/var/lib/kubelet/pods/6da0c03e-6ec9-47c6-aa85-3d49864d1a9b/volumes" Oct 30 19:10:00.646962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.646925 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx"] Oct 30 19:10:00.647496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.647480 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" containerName="revoke-keys" Oct 30 19:10:00.647550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.647497 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" containerName="revoke-keys" Oct 30 19:10:00.647595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.647571 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" containerName="revoke-keys" Oct 30 19:10:00.652041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.652025 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.710918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.710886 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.711100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.710942 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbdk\" (UniqueName: \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.711100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.711069 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.749531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.749498 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:00.749531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.749524 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:00.749727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.749512 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:00.749727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.749498 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:00.752321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.752301 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx"] Oct 30 19:10:00.812470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.812434 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.812675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.812584 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.812675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.812625 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbdk\" (UniqueName: \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.813439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.813409 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.815038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.815012 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.870718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.870686 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbdk\" (UniqueName: \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\") pod \"maas-api-revoke-sub-3f067dc6-pttcx\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:00.965474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:00.965444 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:01.341156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:01.341074 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx"] Oct 30 19:10:01.347566 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:01.347513 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af3be7e_b45a_4bf1_94f8_589de1c92175.slice/crio-98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f WatchSource:0}: Error finding container 98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f: Status 404 returned error can't find the container with id 98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f Oct 30 19:10:01.376005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:01.375967 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" event={"ID":"8af3be7e-b45a-4bf1-94f8-589de1c92175","Type":"ContainerStarted","Data":"98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f"} Oct 30 19:10:02.383812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:02.383771 4789 generic.go:356] "Generic (PLEG): container finished" podID="8af3be7e-b45a-4bf1-94f8-589de1c92175" containerID="c3289c6a806d877c8ba6d50e80cd54dda81981cd1f4f55ea68797ba3119098ff" exitCode=0 Oct 30 19:10:02.384236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:02.383864 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" event={"ID":"8af3be7e-b45a-4bf1-94f8-589de1c92175","Type":"ContainerDied","Data":"c3289c6a806d877c8ba6d50e80cd54dda81981cd1f4f55ea68797ba3119098ff"} Oct 30 19:10:03.538850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.538827 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:03.646295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.646205 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\" (UniqueName: \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\") pod \"8af3be7e-b45a-4bf1-94f8-589de1c92175\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " Oct 30 19:10:03.646446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.646326 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\") pod \"8af3be7e-b45a-4bf1-94f8-589de1c92175\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " Oct 30 19:10:03.646446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.646409 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\") pod \"8af3be7e-b45a-4bf1-94f8-589de1c92175\" (UID: \"8af3be7e-b45a-4bf1-94f8-589de1c92175\") " Oct 30 19:10:03.646786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.646756 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca" pod "8af3be7e-b45a-4bf1-94f8-589de1c92175" (UID: "8af3be7e-b45a-4bf1-94f8-589de1c92175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:03.648455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.648411 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk" pod "8af3be7e-b45a-4bf1-94f8-589de1c92175" (UID: "8af3be7e-b45a-4bf1-94f8-589de1c92175"). InnerVolumeSpecName "kube-api-access-kqbdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:03.648669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.648636 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca" pod "8af3be7e-b45a-4bf1-94f8-589de1c92175" (UID: "8af3be7e-b45a-4bf1-94f8-589de1c92175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:03.748271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.748237 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:03.748271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.748271 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af3be7e-b45a-4bf1-94f8-589de1c92175-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:03.748471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:03.748289 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbdk\" (UniqueName: \"kubernetes.io/projected/8af3be7e-b45a-4bf1-94f8-589de1c92175-kube-api-access-kqbdk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:04.398848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:04.398820 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" Oct 30 19:10:04.399011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:04.398859 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx" event={"ID":"8af3be7e-b45a-4bf1-94f8-589de1c92175","Type":"ContainerDied","Data":"98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f"} Oct 30 19:10:04.399011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:04.398937 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98869683cbbc5260431c698d32f569ddcf9fc93603b345809a20c9545368648f" Oct 30 19:10:11.374976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.374932 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq"] Oct 30 19:10:11.375543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.375522 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8af3be7e-b45a-4bf1-94f8-589de1c92175" containerName="revoke-keys" Oct 30 19:10:11.375605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.375547 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af3be7e-b45a-4bf1-94f8-589de1c92175" containerName="revoke-keys" Oct 30 19:10:11.375723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.375709 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af3be7e-b45a-4bf1-94f8-589de1c92175" containerName="revoke-keys" Oct 30 19:10:11.380860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.380838 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.430051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.430012 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:11.430277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.430235 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:11.430277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.430239 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:11.430410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.430324 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:11.479285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.479257 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq"] Oct 30 19:10:11.534453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.534399 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.534721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.534605 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gpc\" (UniqueName: \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.534721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.534670 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.635434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.635333 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gpc\" (UniqueName: \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.635434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.635395 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.635680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.635456 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.636451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.636421 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.637981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.637958 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.699894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.699860 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gpc\" (UniqueName: \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\") pod \"maas-api-revoke-sub-6a294f63-jmwqq\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:11.993966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:11.993929 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:12.375614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:12.375517 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq"] Oct 30 19:10:12.381825 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:12.381796 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f7e488_4b0c_45fb_accf_e4f21eab6e7c.slice/crio-ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c WatchSource:0}: Error finding container ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c: Status 404 returned error can't find the container with id ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c Oct 30 19:10:12.454497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:12.454464 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" event={"ID":"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c","Type":"ContainerStarted","Data":"ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c"} Oct 30 19:10:13.461731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:13.461693 4789 generic.go:356] "Generic (PLEG): container finished" podID="a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" containerID="014cb11eb31a1e15973c106cac679c825949494be63da03ab56f3c2c11a66b89" exitCode=0 Oct 30 19:10:13.462115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:13.461730 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" event={"ID":"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c","Type":"ContainerDied","Data":"014cb11eb31a1e15973c106cac679c825949494be63da03ab56f3c2c11a66b89"} Oct 30 19:10:14.604316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.604289 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:14.768052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.767974 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\" (UniqueName: \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\") pod \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " Oct 30 19:10:14.768190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.768061 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\") pod \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " Oct 30 19:10:14.768190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.768177 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\") pod \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\" (UID: \"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c\") " Oct 30 19:10:14.768574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.768539 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca" pod "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" (UID: "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:14.770628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.770599 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca" pod "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" (UID: "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:14.770628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.770618 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc" pod "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" (UID: "a1f7e488-4b0c-45fb-accf-e4f21eab6e7c"). InnerVolumeSpecName "kube-api-access-f4gpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:14.869646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.869613 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gpc\" (UniqueName: \"kubernetes.io/projected/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-kube-api-access-f4gpc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:14.869646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.869643 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:14.869646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:14.869653 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:15.050831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.050732 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k"] Oct 30 19:10:15.053369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.053342 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0290e228-dfx5k"] Oct 30 19:10:15.474776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.474746 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" Oct 30 19:10:15.474974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.474773 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq" event={"ID":"a1f7e488-4b0c-45fb-accf-e4f21eab6e7c","Type":"ContainerDied","Data":"ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c"} Oct 30 19:10:15.474974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.474823 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee7726cd430b9ec17db5346c8ab5787b1bb2046522ebcba125d781833c846f8c" Oct 30 19:10:15.667621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:15.667589 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4146c77f-f264-4470-9b08-43caec4041b7" path="/var/lib/kubelet/pods/4146c77f-f264-4470-9b08-43caec4041b7/volumes" Oct 30 19:10:16.413014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.412979 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr"] Oct 30 19:10:16.413600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.413572 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" containerName="revoke-keys" Oct 30 19:10:16.413600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.413597 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" containerName="revoke-keys" Oct 30 19:10:16.413756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.413744 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" containerName="revoke-keys" Oct 30 19:10:16.417057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.417040 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.468513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.468478 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:16.468716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.468601 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:16.514802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.514766 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:16.514999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.514880 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:16.517575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.517553 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr"] Oct 30 19:10:16.586625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.586586 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.586818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.586648 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmrr\" (UniqueName: \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.586818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.586754 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.688324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.688288 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.688743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.688434 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.688743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.688477 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmrr\" (UniqueName: \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.689175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.689155 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.690633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.690608 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:16.746419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:16.746382 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmrr\" (UniqueName: \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\") pod \"maas-api-revoke-sub-152d03dd-b92wr\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:17.030293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:17.030195 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:17.407722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:17.407638 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr"] Oct 30 19:10:17.489596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:17.489559 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" event={"ID":"55015f43-727e-4bc7-a4d1-ca2157a507ab","Type":"ContainerStarted","Data":"846857f7e57d090ecfddde2975f43520060343aac48d526ed9a4df719dfc08ff"} Oct 30 19:10:18.196855 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:10:18.196805 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:10:18.498794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:18.498708 4789 generic.go:356] "Generic (PLEG): container finished" podID="55015f43-727e-4bc7-a4d1-ca2157a507ab" containerID="afd773c6cc746154bdff30a89a8579afc8a7fd49ca32e71e732a2c2b9d98daf4" exitCode=0 Oct 30 19:10:18.498951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:18.498794 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" event={"ID":"55015f43-727e-4bc7-a4d1-ca2157a507ab","Type":"ContainerDied","Data":"afd773c6cc746154bdff30a89a8579afc8a7fd49ca32e71e732a2c2b9d98daf4"} Oct 30 19:10:19.645674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.645646 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:19.822457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.822372 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\" (UniqueName: \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\") pod \"55015f43-727e-4bc7-a4d1-ca2157a507ab\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " Oct 30 19:10:19.822457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.822421 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\") pod \"55015f43-727e-4bc7-a4d1-ca2157a507ab\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " Oct 30 19:10:19.822691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.822608 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\") pod \"55015f43-727e-4bc7-a4d1-ca2157a507ab\" (UID: \"55015f43-727e-4bc7-a4d1-ca2157a507ab\") " Oct 30 19:10:19.823051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.823006 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca" pod "55015f43-727e-4bc7-a4d1-ca2157a507ab" (UID: "55015f43-727e-4bc7-a4d1-ca2157a507ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:19.824714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.824686 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca" pod "55015f43-727e-4bc7-a4d1-ca2157a507ab" (UID: "55015f43-727e-4bc7-a4d1-ca2157a507ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:19.824804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.824741 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr" pod "55015f43-727e-4bc7-a4d1-ca2157a507ab" (UID: "55015f43-727e-4bc7-a4d1-ca2157a507ab"). InnerVolumeSpecName "kube-api-access-qxmrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:19.924257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.924190 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxmrr\" (UniqueName: \"kubernetes.io/projected/55015f43-727e-4bc7-a4d1-ca2157a507ab-kube-api-access-qxmrr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:19.924257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.924256 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:19.924505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:19.924273 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55015f43-727e-4bc7-a4d1-ca2157a507ab-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:20.511465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.511421 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" event={"ID":"55015f43-727e-4bc7-a4d1-ca2157a507ab","Type":"ContainerDied","Data":"846857f7e57d090ecfddde2975f43520060343aac48d526ed9a4df719dfc08ff"} Oct 30 19:10:20.511633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.511473 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846857f7e57d090ecfddde2975f43520060343aac48d526ed9a4df719dfc08ff" Oct 30 19:10:20.511633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.511432 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr" Oct 30 19:10:20.515243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.515204 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 19:10:20.515714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.515698 4789 cpu_manager.go:414] "containerMap: removing container" podUID="55015f43-727e-4bc7-a4d1-ca2157a507ab" containerName="revoke-keys" Oct 30 19:10:20.515714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.515714 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55015f43-727e-4bc7-a4d1-ca2157a507ab" containerName="revoke-keys" Oct 30 19:10:20.515851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.515796 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="55015f43-727e-4bc7-a4d1-ca2157a507ab" containerName="revoke-keys" Oct 30 19:10:20.519185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.519165 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.614225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.614182 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-330025\"/\"distinct-2-w2-330025-kserve-self-signed-certs\"" Oct 30 19:10:20.617083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.617062 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 19:10:20.631234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631177 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.631389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631312 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.631389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631375 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.631496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631409 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.631577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631561 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.631688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.631671 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9xb\" (UniqueName: \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.732665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732633 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732735 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9xb\" (UniqueName: \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732806 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732854 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732908 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.732943 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.733073 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.733201 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.733662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.733482 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.735369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.735343 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.736032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.736009 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.797815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.797739 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9xb\" (UniqueName: \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\") pod \"distinct-2-w2-330025-kserve-78987cff7f-czss5\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.802111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.802081 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c"] Oct 30 19:10:20.805849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.805830 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:20.831384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.831353 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:20.862605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.862350 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c"] Oct 30 19:10:20.862760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.862656 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:20.863448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.863341 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:20.863448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.863395 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:20.863839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.863816 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:20.949957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.937014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c676h\" (UniqueName: \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:20.949957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.937065 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:20.949957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:20.937233 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.038388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.037897 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.038388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.037998 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c676h\" (UniqueName: \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.038388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.038034 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.038784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.038763 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.040485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.040461 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.096160 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.096082 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c676h\" (UniqueName: \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\") pod \"maas-api-revoke-sub-12505fe0-t9t7c\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.129886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.129851 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:21.256859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.256825 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 19:10:21.262775 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:21.262743 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2db97b9_2af8_4f06_84d4_3cff4a5854ac.slice/crio-0e6fecb6f5326303c7181d9b61eeb671a69acc485f2f03feb1ad2ab47c57be03 WatchSource:0}: Error finding container 0e6fecb6f5326303c7181d9b61eeb671a69acc485f2f03feb1ad2ab47c57be03: Status 404 returned error can't find the container with id 0e6fecb6f5326303c7181d9b61eeb671a69acc485f2f03feb1ad2ab47c57be03 Oct 30 19:10:21.508702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.508671 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c"] Oct 30 19:10:21.514581 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:21.514557 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1be968d_8797_4ccb_b0c9_881ced34c51a.slice/crio-c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9 WatchSource:0}: Error finding container c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9: Status 404 returned error can't find the container with id c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9 Oct 30 19:10:21.519120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.519097 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" event={"ID":"c2db97b9-2af8-4f06-84d4-3cff4a5854ac","Type":"ContainerStarted","Data":"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5"} Oct 30 19:10:21.519239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.519134 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" event={"ID":"c2db97b9-2af8-4f06-84d4-3cff4a5854ac","Type":"ContainerStarted","Data":"0e6fecb6f5326303c7181d9b61eeb671a69acc485f2f03feb1ad2ab47c57be03"} Oct 30 19:10:21.632224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.632168 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" podStartSLOduration=1.632154871 podStartE2EDuration="1.632154871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:21.630735029 +0000 UTC m=+1458.132645514" watchObservedRunningTime="2026-10-30 19:10:21.632154871 +0000 UTC m=+1458.134065356" Oct 30 19:10:21.832330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:21.832284 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:22.527564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:22.527530 4789 generic.go:356] "Generic (PLEG): container finished" podID="e1be968d-8797-4ccb-b0c9-881ced34c51a" containerID="68f38e5eb4918196caa915cb8c758e37088efb25ea5771435cef252f52c17ada" exitCode=0 Oct 30 19:10:22.527734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:22.527621 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" event={"ID":"e1be968d-8797-4ccb-b0c9-881ced34c51a","Type":"ContainerDied","Data":"68f38e5eb4918196caa915cb8c758e37088efb25ea5771435cef252f52c17ada"} Oct 30 19:10:22.527734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:22.527675 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" event={"ID":"e1be968d-8797-4ccb-b0c9-881ced34c51a","Type":"ContainerStarted","Data":"c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9"} Oct 30 19:10:23.049099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.049067 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7"] Oct 30 19:10:23.053791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.053757 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7c0445b-ffbl7"] Oct 30 19:10:23.670659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.670631 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a528eea-eacd-49f4-8047-948171ab4b55" path="/var/lib/kubelet/pods/0a528eea-eacd-49f4-8047-948171ab4b55/volumes" Oct 30 19:10:23.683863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.683836 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:23.871861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.871785 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\") pod \"e1be968d-8797-4ccb-b0c9-881ced34c51a\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " Oct 30 19:10:23.871861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.871851 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\" (UniqueName: \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\") pod \"e1be968d-8797-4ccb-b0c9-881ced34c51a\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " Oct 30 19:10:23.872136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.872053 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\") pod \"e1be968d-8797-4ccb-b0c9-881ced34c51a\" (UID: \"e1be968d-8797-4ccb-b0c9-881ced34c51a\") " Oct 30 19:10:23.872439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.872415 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca" pod "e1be968d-8797-4ccb-b0c9-881ced34c51a" (UID: "e1be968d-8797-4ccb-b0c9-881ced34c51a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:23.873826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.873805 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca" pod "e1be968d-8797-4ccb-b0c9-881ced34c51a" (UID: "e1be968d-8797-4ccb-b0c9-881ced34c51a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:23.874009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.873985 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h" pod "e1be968d-8797-4ccb-b0c9-881ced34c51a" (UID: "e1be968d-8797-4ccb-b0c9-881ced34c51a"). InnerVolumeSpecName "kube-api-access-c676h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:23.973528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.973494 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:23.973528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.973523 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1be968d-8797-4ccb-b0c9-881ced34c51a-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:23.973528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:23.973532 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c676h\" (UniqueName: \"kubernetes.io/projected/e1be968d-8797-4ccb-b0c9-881ced34c51a-kube-api-access-c676h\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:24.542384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:24.542343 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" Oct 30 19:10:24.542384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:24.542353 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c" event={"ID":"e1be968d-8797-4ccb-b0c9-881ced34c51a","Type":"ContainerDied","Data":"c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9"} Oct 30 19:10:24.542812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:24.542405 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c501a5fbfc850c7f6409cee820f2dae35c35a9785b4781dac28bbc5b136d69a9" Oct 30 19:10:28.636373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.636339 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8"] Oct 30 19:10:28.636873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.636858 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e1be968d-8797-4ccb-b0c9-881ced34c51a" containerName="revoke-keys" Oct 30 19:10:28.636942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.636875 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1be968d-8797-4ccb-b0c9-881ced34c51a" containerName="revoke-keys" Oct 30 19:10:28.636994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.636985 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1be968d-8797-4ccb-b0c9-881ced34c51a" containerName="revoke-keys" Oct 30 19:10:28.640529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.640505 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.692796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.692749 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:28.726634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.726596 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8559\" (UniqueName: \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.726820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.726706 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.726893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.726829 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.738798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.738773 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:28.738951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.738801 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:28.738951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.738804 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:28.741739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.741716 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8"] Oct 30 19:10:28.828356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.828323 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.828565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.828408 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.828565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.828461 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8559\" (UniqueName: \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.829026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.829005 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.831348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.831320 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.887964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.887895 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8559\" (UniqueName: \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\") pod \"maas-api-revoke-sub-f404c535-m57b8\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:28.956043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:28.956008 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:29.352330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:29.352288 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8"] Oct 30 19:10:29.357478 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:29.357440 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb42651_c5e6_47fd_90a2_5f46a102e114.slice/crio-c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831 WatchSource:0}: Error finding container c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831: Status 404 returned error can't find the container with id c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831 Oct 30 19:10:29.577098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:29.577055 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" event={"ID":"feb42651-c5e6-47fd-90a2-5f46a102e114","Type":"ContainerStarted","Data":"c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831"} Oct 30 19:10:30.587116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:30.587076 4789 generic.go:356] "Generic (PLEG): container finished" podID="feb42651-c5e6-47fd-90a2-5f46a102e114" containerID="6791e6329f332b7d8946e6ebafe1d4914d5e8f55ca3a975fd3d7b036fb38056d" exitCode=0 Oct 30 19:10:30.587627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:30.587160 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" event={"ID":"feb42651-c5e6-47fd-90a2-5f46a102e114","Type":"ContainerDied","Data":"6791e6329f332b7d8946e6ebafe1d4914d5e8f55ca3a975fd3d7b036fb38056d"} Oct 30 19:10:30.832602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:30.832510 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:30.845406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:30.845376 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:31.610652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.610619 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 19:10:31.763367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.763343 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:31.862507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.862429 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\") pod \"feb42651-c5e6-47fd-90a2-5f46a102e114\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " Oct 30 19:10:31.862507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.862492 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\" (UniqueName: \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\") pod \"feb42651-c5e6-47fd-90a2-5f46a102e114\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " Oct 30 19:10:31.862692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.862543 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\") pod \"feb42651-c5e6-47fd-90a2-5f46a102e114\" (UID: \"feb42651-c5e6-47fd-90a2-5f46a102e114\") " Oct 30 19:10:31.862870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.862846 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca" pod "feb42651-c5e6-47fd-90a2-5f46a102e114" (UID: "feb42651-c5e6-47fd-90a2-5f46a102e114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:31.864611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.864587 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559" pod "feb42651-c5e6-47fd-90a2-5f46a102e114" (UID: "feb42651-c5e6-47fd-90a2-5f46a102e114"). InnerVolumeSpecName "kube-api-access-f8559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:31.864611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.864603 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca" pod "feb42651-c5e6-47fd-90a2-5f46a102e114" (UID: "feb42651-c5e6-47fd-90a2-5f46a102e114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:31.963574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.963537 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8559\" (UniqueName: \"kubernetes.io/projected/feb42651-c5e6-47fd-90a2-5f46a102e114-kube-api-access-f8559\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:31.963574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.963567 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:31.963574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:31.963580 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb42651-c5e6-47fd-90a2-5f46a102e114-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:32.604288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:32.604247 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" event={"ID":"feb42651-c5e6-47fd-90a2-5f46a102e114","Type":"ContainerDied","Data":"c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831"} Oct 30 19:10:32.604288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:32.604284 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8" Oct 30 19:10:32.604509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:32.604315 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3928589d0c10f296def40e42ef732aa3bec3169a4b2dff0a0f5f3ca2edc7831" Oct 30 19:10:34.048302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:34.048262 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd"] Oct 30 19:10:34.094793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:34.094760 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4691fea4-5wdvd"] Oct 30 19:10:35.171044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.171004 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp"] Oct 30 19:10:35.171746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.171722 4789 cpu_manager.go:414] "containerMap: removing container" podUID="feb42651-c5e6-47fd-90a2-5f46a102e114" containerName="revoke-keys" Oct 30 19:10:35.171816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.171747 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb42651-c5e6-47fd-90a2-5f46a102e114" containerName="revoke-keys" Oct 30 19:10:35.171914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.171899 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb42651-c5e6-47fd-90a2-5f46a102e114" containerName="revoke-keys" Oct 30 19:10:35.175820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.175792 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.198729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.198692 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.198853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.198766 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.198912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.198874 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hzxx\" (UniqueName: \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.227353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.227204 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:35.273324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.273289 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:35.273503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.273328 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:35.273559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.273543 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:35.276385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.276359 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp"] Oct 30 19:10:35.299625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.299596 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hzxx\" (UniqueName: \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.299760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.299722 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.299813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.299759 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.300778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.300758 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.302241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.302198 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.360106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.360074 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hzxx\" (UniqueName: \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\") pod \"maas-api-revoke-sub-5ed6cc2c-n29qp\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.489402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.489364 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:35.667295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.667263 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400a8996-d6e5-4663-bdab-1273765d4eba" path="/var/lib/kubelet/pods/400a8996-d6e5-4663-bdab-1273765d4eba/volumes" Oct 30 19:10:35.863648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:35.863569 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp"] Oct 30 19:10:35.875635 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:35.875602 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9430819e_f32c_43c0_a286_36c54ac3d6da.slice/crio-bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4 WatchSource:0}: Error finding container bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4: Status 404 returned error can't find the container with id bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4 Oct 30 19:10:36.632604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:36.632513 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" event={"ID":"9430819e-f32c-43c0-a286-36c54ac3d6da","Type":"ContainerStarted","Data":"0a81938a00223d31ac0ed42a598f361edfdd7fc2b0d7c80f3e9d140cabb31b59"} Oct 30 19:10:36.632604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:36.632564 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" event={"ID":"9430819e-f32c-43c0-a286-36c54ac3d6da","Type":"ContainerStarted","Data":"bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4"} Oct 30 19:10:36.745766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:36.745708 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" podStartSLOduration=1.745690475 podStartE2EDuration="1.745690475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:36.742356986 +0000 UTC m=+1473.244267484" watchObservedRunningTime="2026-10-30 19:10:36.745690475 +0000 UTC m=+1473.247600962" Oct 30 19:10:37.640916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:37.640880 4789 generic.go:356] "Generic (PLEG): container finished" podID="9430819e-f32c-43c0-a286-36c54ac3d6da" containerID="0a81938a00223d31ac0ed42a598f361edfdd7fc2b0d7c80f3e9d140cabb31b59" exitCode=0 Oct 30 19:10:37.641372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:37.640967 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" event={"ID":"9430819e-f32c-43c0-a286-36c54ac3d6da","Type":"ContainerDied","Data":"0a81938a00223d31ac0ed42a598f361edfdd7fc2b0d7c80f3e9d140cabb31b59"} Oct 30 19:10:38.782570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.782538 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:38.838012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.837974 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\") pod \"9430819e-f32c-43c0-a286-36c54ac3d6da\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " Oct 30 19:10:38.838170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.838035 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\") pod \"9430819e-f32c-43c0-a286-36c54ac3d6da\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " Oct 30 19:10:38.838170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.838117 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\" (UniqueName: \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\") pod \"9430819e-f32c-43c0-a286-36c54ac3d6da\" (UID: \"9430819e-f32c-43c0-a286-36c54ac3d6da\") " Oct 30 19:10:38.838426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.838398 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca" pod "9430819e-f32c-43c0-a286-36c54ac3d6da" (UID: "9430819e-f32c-43c0-a286-36c54ac3d6da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:38.840244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.840204 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx" pod "9430819e-f32c-43c0-a286-36c54ac3d6da" (UID: "9430819e-f32c-43c0-a286-36c54ac3d6da"). InnerVolumeSpecName "kube-api-access-9hzxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:38.840244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.840226 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca" pod "9430819e-f32c-43c0-a286-36c54ac3d6da" (UID: "9430819e-f32c-43c0-a286-36c54ac3d6da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:38.939652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.939620 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:38.939826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.939657 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9430819e-f32c-43c0-a286-36c54ac3d6da-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:38.939826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:38.939674 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hzxx\" (UniqueName: \"kubernetes.io/projected/9430819e-f32c-43c0-a286-36c54ac3d6da-kube-api-access-9hzxx\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:39.421143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.421047 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h"] Oct 30 19:10:39.421622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.421605 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9430819e-f32c-43c0-a286-36c54ac3d6da" containerName="revoke-keys" Oct 30 19:10:39.421682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.421623 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9430819e-f32c-43c0-a286-36c54ac3d6da" containerName="revoke-keys" Oct 30 19:10:39.421731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.421717 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9430819e-f32c-43c0-a286-36c54ac3d6da" containerName="revoke-keys" Oct 30 19:10:39.424806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.424784 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.481094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.481057 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h"] Oct 30 19:10:39.545280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.545244 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95tl\" (UniqueName: \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.545456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.545380 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.545456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.545421 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.646347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.646307 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.646547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.646361 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.646547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.646455 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95tl\" (UniqueName: \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.647104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.647079 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.648908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.648883 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.653547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.653512 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" Oct 30 19:10:39.653651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.653540 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp" event={"ID":"9430819e-f32c-43c0-a286-36c54ac3d6da","Type":"ContainerDied","Data":"bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4"} Oct 30 19:10:39.653651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.653597 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbab881898d0d81f480e275c465b548da6f6da0c030394e2eea32a0737259d4" Oct 30 19:10:39.704849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.704819 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95tl\" (UniqueName: \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\") pod \"maas-api-revoke-sub-f367b4ee-mnw5h\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:39.737749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:39.737723 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:40.119883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:40.119800 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h"] Oct 30 19:10:40.125303 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:40.125266 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bacce0_beb9_4e1e_853e_622f651e4d7d.slice/crio-111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94 WatchSource:0}: Error finding container 111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94: Status 404 returned error can't find the container with id 111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94 Oct 30 19:10:40.661719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:40.661618 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" event={"ID":"22bacce0-beb9-4e1e-853e-622f651e4d7d","Type":"ContainerStarted","Data":"ed79cb9663ad0d15dbd636149c03f2eb84de4ea30c35d86d817bd94553e47433"} Oct 30 19:10:40.661719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:40.661683 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" event={"ID":"22bacce0-beb9-4e1e-853e-622f651e4d7d","Type":"ContainerStarted","Data":"111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94"} Oct 30 19:10:40.771643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:40.771592 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" podStartSLOduration=1.771577772 podStartE2EDuration="1.771577772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:40.770728167 +0000 UTC m=+1477.272638654" watchObservedRunningTime="2026-10-30 19:10:40.771577772 +0000 UTC m=+1477.273488257" Oct 30 19:10:41.669405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:41.669368 4789 generic.go:356] "Generic (PLEG): container finished" podID="22bacce0-beb9-4e1e-853e-622f651e4d7d" containerID="ed79cb9663ad0d15dbd636149c03f2eb84de4ea30c35d86d817bd94553e47433" exitCode=0 Oct 30 19:10:41.669888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:41.669450 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" event={"ID":"22bacce0-beb9-4e1e-853e-622f651e4d7d","Type":"ContainerDied","Data":"ed79cb9663ad0d15dbd636149c03f2eb84de4ea30c35d86d817bd94553e47433"} Oct 30 19:10:42.835072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.835041 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:42.880851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.880801 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\") pod \"22bacce0-beb9-4e1e-853e-622f651e4d7d\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " Oct 30 19:10:42.881012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.880871 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\" (UniqueName: \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\") pod \"22bacce0-beb9-4e1e-853e-622f651e4d7d\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " Oct 30 19:10:42.881012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.880905 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\") pod \"22bacce0-beb9-4e1e-853e-622f651e4d7d\" (UID: \"22bacce0-beb9-4e1e-853e-622f651e4d7d\") " Oct 30 19:10:42.881332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.881306 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca" pod "22bacce0-beb9-4e1e-853e-622f651e4d7d" (UID: "22bacce0-beb9-4e1e-853e-622f651e4d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:42.881446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.881428 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:42.882893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.882870 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca" pod "22bacce0-beb9-4e1e-853e-622f651e4d7d" (UID: "22bacce0-beb9-4e1e-853e-622f651e4d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:42.883021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.883002 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl" pod "22bacce0-beb9-4e1e-853e-622f651e4d7d" (UID: "22bacce0-beb9-4e1e-853e-622f651e4d7d"). InnerVolumeSpecName "kube-api-access-p95tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:42.982868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.982829 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22bacce0-beb9-4e1e-853e-622f651e4d7d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:42.982868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:42.982860 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95tl\" (UniqueName: \"kubernetes.io/projected/22bacce0-beb9-4e1e-853e-622f651e4d7d-kube-api-access-p95tl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:43.683198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.683169 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" Oct 30 19:10:43.683198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.683196 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h" event={"ID":"22bacce0-beb9-4e1e-853e-622f651e4d7d","Type":"ContainerDied","Data":"111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94"} Oct 30 19:10:43.683394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.683260 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="111cd33063f42aa0153e9d1c9199f8c3037154b3c968301f04d93083a4c95e94" Oct 30 19:10:43.817051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.817009 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:10:43.817686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.817653 4789 cpu_manager.go:414] "containerMap: removing container" podUID="22bacce0-beb9-4e1e-853e-622f651e4d7d" containerName="revoke-keys" Oct 30 19:10:43.817686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.817680 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bacce0-beb9-4e1e-853e-622f651e4d7d" containerName="revoke-keys" Oct 30 19:10:43.817825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.817820 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bacce0-beb9-4e1e-853e-622f651e4d7d" containerName="revoke-keys" Oct 30 19:10:43.821399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.821379 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892085 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sx7\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892240 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892281 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892315 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892341 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892394 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892434 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892480 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.892603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.892517 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994248 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994311 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994348 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994428 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994471 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994516 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994551 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994590 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sx7\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.994923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.994843 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.995429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.995306 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.995429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.995374 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.995602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.995435 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.995602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.995433 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.996858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.996836 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:43.997643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:43.997622 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:44.020572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.020542 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:10:44.053685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.053653 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:44.054032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.054003 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sx7\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:44.134859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.134822 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:44.577274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.577237 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:10:44.581643 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:44.581617 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8945dc4_58b8_4998_aa6b_48d548bd53a1.slice/crio-bf5b7adae87bb7811da0bb74b64e10b286d90a6bc259b580fd71906f2ab3f178 WatchSource:0}: Error finding container bf5b7adae87bb7811da0bb74b64e10b286d90a6bc259b580fd71906f2ab3f178: Status 404 returned error can't find the container with id bf5b7adae87bb7811da0bb74b64e10b286d90a6bc259b580fd71906f2ab3f178 Oct 30 19:10:44.583892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.583863 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:44.583956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.583927 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:44.583999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.583956 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:44.697945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.697906 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" event={"ID":"b8945dc4-58b8-4998-aa6b-48d548bd53a1","Type":"ContainerStarted","Data":"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643"} Oct 30 19:10:44.698137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.697974 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" event={"ID":"b8945dc4-58b8-4998-aa6b-48d548bd53a1","Type":"ContainerStarted","Data":"bf5b7adae87bb7811da0bb74b64e10b286d90a6bc259b580fd71906f2ab3f178"} Oct 30 19:10:44.815183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.815129 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podStartSLOduration=1.8151149389999999 podStartE2EDuration="1.815114939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:44.813652265 +0000 UTC m=+1481.315562750" watchObservedRunningTime="2026-10-30 19:10:44.815114939 +0000 UTC m=+1481.317025425" Oct 30 19:10:44.870074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.869997 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:10:44.874888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.874867 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:44.923892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.923861 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9275c3-openshift-default-dockercfg-hq6vr\"" Oct 30 19:10:44.932635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:44.931086 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:10:45.006172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006052 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006484 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006653 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqv4t\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006825 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006861 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006899 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.006971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006943 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.007008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.006981 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.007048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.007011 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.051314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.051268 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z"] Oct 30 19:10:45.052901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.052872 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d87ebfad-bgq8z"] Oct 30 19:10:45.108194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108152 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108241 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108295 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108338 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108376 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108406 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108481 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108537 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108587 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqv4t\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.108823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.108645 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.109254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.109231 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.109535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.109506 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.109700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.109678 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.109832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.109804 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.111170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.111142 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.112527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.112494 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.135439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.135352 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:45.137078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.137014 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:45.137197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.137122 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:45.166673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.166641 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.167149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.167130 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqv4t\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.188814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.188790 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:45.633990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.633954 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:10:45.640817 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:45.640790 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fec5d34_5588_4f06_96ad_447feba5391b.slice/crio-35a8a76c73e8d38a717336579b18d07d948fc1e539be46b35411b903b5090540 WatchSource:0}: Error finding container 35a8a76c73e8d38a717336579b18d07d948fc1e539be46b35411b903b5090540: Status 404 returned error can't find the container with id 35a8a76c73e8d38a717336579b18d07d948fc1e539be46b35411b903b5090540 Oct 30 19:10:45.643517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.643484 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:45.643604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.643558 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:45.643604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.643586 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:45.668721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.668693 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1" path="/var/lib/kubelet/pods/c2ff7041-ecf0-45d1-ab2b-38c35aeac3c1/volumes" Oct 30 19:10:45.731453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:45.731357 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" event={"ID":"5fec5d34-5588-4f06-96ad-447feba5391b","Type":"ContainerStarted","Data":"35a8a76c73e8d38a717336579b18d07d948fc1e539be46b35411b903b5090540"} Oct 30 19:10:46.112236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.111391 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:10:46.120316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.118469 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.136987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.136920 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:46.136987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.136968 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:46.238759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.238714 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.238971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.238774 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.238971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.238903 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.238971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.238939 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.239163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.239012 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.239163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.239044 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.239163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.239096 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.239163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.239123 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l677h\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.239163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.239156 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.242642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.242615 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:10:46.340240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340171 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340238 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340268 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340297 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l677h\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340329 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340355 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340394 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340560 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340612 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.340956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.340923 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.341051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.341023 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.341656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.341601 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.341656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.341632 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.341656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.341647 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.343139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.343113 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.343449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.343427 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.408164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.408051 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.427029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.426963 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l677h\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.454651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.444065 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:46.764306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.763742 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" event={"ID":"5fec5d34-5588-4f06-96ad-447feba5391b","Type":"ContainerStarted","Data":"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d"} Oct 30 19:10:46.885983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:46.885929 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podStartSLOduration=2.885913796 podStartE2EDuration="2.885913796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:46.883173191 +0000 UTC m=+1483.385083676" watchObservedRunningTime="2026-10-30 19:10:46.885913796 +0000 UTC m=+1483.387824282" Oct 30 19:10:47.033286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.032193 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:10:47.065437 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:10:47.065387 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cff649_d727_4d15_88a9_3fabb4833514.slice/crio-7d74e5f700f5c9e6291447fcde9ab214d0973c12742d342ec76e4a29137cf688 WatchSource:0}: Error finding container 7d74e5f700f5c9e6291447fcde9ab214d0973c12742d342ec76e4a29137cf688: Status 404 returned error can't find the container with id 7d74e5f700f5c9e6291447fcde9ab214d0973c12742d342ec76e4a29137cf688 Oct 30 19:10:47.068171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.068142 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:47.068250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.068226 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:47.068286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.068264 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:10:47.137233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.136558 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:47.137233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.136624 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:47.190189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.190164 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:47.195405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.195361 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:47.195657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.195603 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:47.774496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.774463 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" event={"ID":"98cff649-d727-4d15-88a9-3fabb4833514","Type":"ContainerStarted","Data":"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb"} Oct 30 19:10:47.774662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.774516 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" event={"ID":"98cff649-d727-4d15-88a9-3fabb4833514","Type":"ContainerStarted","Data":"7d74e5f700f5c9e6291447fcde9ab214d0973c12742d342ec76e4a29137cf688"} Oct 30 19:10:47.908806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:47.903121 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podStartSLOduration=1.9031016630000002 podStartE2EDuration="1.903101663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:47.901693512 +0000 UTC m=+1484.403603998" watchObservedRunningTime="2026-10-30 19:10:47.903101663 +0000 UTC m=+1484.405012150" Oct 30 19:10:48.141318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.135642 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:48.141318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.135711 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:48.190090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.189993 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:48.190090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.190040 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:48.281547 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:10:48.281434 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:10:48.449121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.449094 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:48.449569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.449546 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:48.449788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:48.449751 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:49.141232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.138575 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:49.141232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.138639 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:49.191242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.190603 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:49.191242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.190671 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:49.457234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.445417 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:49.457234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:49.445475 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:50.140233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.139561 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:50.140233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.139628 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:50.189781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.189612 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:50.189781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.189682 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:50.456233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.448727 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:50.456233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:50.448786 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:51.136234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.135596 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:51.136234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.135662 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:51.195279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.193393 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:51.195279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.193448 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:51.444935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.444511 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:51.444935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:51.444569 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:52.137670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.137634 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:10:52.137963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.137915 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 19:10:52.189734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.189696 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:52.189897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.189754 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:52.446827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.446794 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:52.447638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:52.447610 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:53.143849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.142472 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:53.143849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.143707 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:53.144826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.144770 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:10:53.190422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.190390 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:10:53.190649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.190622 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:10:53.452517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.452482 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:53.453064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:53.453034 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:54.444452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:54.444420 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:10:54.444648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:54.444494 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:10:55.118125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.118092 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:55.118785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.118514 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:55.119323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.119297 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:10:55.919957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.919862 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg"] Oct 30 19:10:55.923783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.923762 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:55.974813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.974781 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:55.974954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.974903 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:10:55.974954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.974908 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:10:55.974954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.974926 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:10:55.995488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.995455 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:55.995709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.995685 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79lm5\" (UniqueName: \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:55.995887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:55.995863 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.010658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.010633 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg"] Oct 30 19:10:56.097460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.097428 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.097623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.097489 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.097692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.097618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79lm5\" (UniqueName: \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.098302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.098267 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.099967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.099941 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.137819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.137794 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:56.138383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.138363 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:56.139434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.139410 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:10:56.156272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.156243 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79lm5\" (UniqueName: \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\") pod \"maas-api-revoke-sub-9898de95-jngjg\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.238099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.238077 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:56.658606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.658502 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg"] Oct 30 19:10:56.960786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:56.960755 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" event={"ID":"d16211ba-eac8-4c04-a0a4-4ccff634ce35","Type":"ContainerStarted","Data":"e6f30382d8be0ee2f2f46939817dd61fe045e7bb0bb4fc2f271b9216b3c81957"} Oct 30 19:10:57.968509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:57.968471 4789 generic.go:356] "Generic (PLEG): container finished" podID="d16211ba-eac8-4c04-a0a4-4ccff634ce35" containerID="dc5b2c101b3992cd0298d0c6223273e871c8421a3374ffd2ce4d7c636ba8b842" exitCode=0 Oct 30 19:10:57.968867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:57.968507 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" event={"ID":"d16211ba-eac8-4c04-a0a4-4ccff634ce35","Type":"ContainerDied","Data":"dc5b2c101b3992cd0298d0c6223273e871c8421a3374ffd2ce4d7c636ba8b842"} Oct 30 19:10:59.142702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.142674 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:59.233987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.233913 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\") pod \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " Oct 30 19:10:59.234126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.234031 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\" (UniqueName: \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\") pod \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " Oct 30 19:10:59.234197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.234131 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\") pod \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\" (UID: \"d16211ba-eac8-4c04-a0a4-4ccff634ce35\") " Oct 30 19:10:59.234382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.234329 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca" pod "d16211ba-eac8-4c04-a0a4-4ccff634ce35" (UID: "d16211ba-eac8-4c04-a0a4-4ccff634ce35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:59.234713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.234690 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:59.236295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.236233 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5" pod "d16211ba-eac8-4c04-a0a4-4ccff634ce35" (UID: "d16211ba-eac8-4c04-a0a4-4ccff634ce35"). InnerVolumeSpecName "kube-api-access-79lm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:59.236583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.236558 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca" pod "d16211ba-eac8-4c04-a0a4-4ccff634ce35" (UID: "d16211ba-eac8-4c04-a0a4-4ccff634ce35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:59.336252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.336194 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16211ba-eac8-4c04-a0a4-4ccff634ce35-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:59.336252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.336244 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79lm5\" (UniqueName: \"kubernetes.io/projected/d16211ba-eac8-4c04-a0a4-4ccff634ce35-kube-api-access-79lm5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:59.984754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.984718 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" Oct 30 19:10:59.984754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.984734 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg" event={"ID":"d16211ba-eac8-4c04-a0a4-4ccff634ce35","Type":"ContainerDied","Data":"e6f30382d8be0ee2f2f46939817dd61fe045e7bb0bb4fc2f271b9216b3c81957"} Oct 30 19:10:59.984953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:10:59.984788 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f30382d8be0ee2f2f46939817dd61fe045e7bb0bb4fc2f271b9216b3c81957" Oct 30 19:11:02.578554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.578513 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns"] Oct 30 19:11:02.579256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.579237 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d16211ba-eac8-4c04-a0a4-4ccff634ce35" containerName="revoke-keys" Oct 30 19:11:02.579256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.579257 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16211ba-eac8-4c04-a0a4-4ccff634ce35" containerName="revoke-keys" Oct 30 19:11:02.579416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.579405 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16211ba-eac8-4c04-a0a4-4ccff634ce35" containerName="revoke-keys" Oct 30 19:11:02.583453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.583427 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.632563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.632529 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:02.632765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.632679 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:02.671109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.671074 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.671319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.671165 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gwr\" (UniqueName: \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.671457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.671425 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.678981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.678948 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:02.678981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.678948 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:02.683885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.683862 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns"] Oct 30 19:11:02.772335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.772297 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.772517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.772392 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gwr\" (UniqueName: \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.772573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.772515 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.773392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.773367 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.774936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.774909 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.831046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.830962 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gwr\" (UniqueName: \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\") pod \"maas-api-revoke-sub-fb022906-hrfns\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:02.905242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:02.903674 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:03.359258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:03.358933 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns"] Oct 30 19:11:03.362949 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:03.362922 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc769b1d_710b_4225_974e_699467d4b70c.slice/crio-5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40 WatchSource:0}: Error finding container 5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40: Status 404 returned error can't find the container with id 5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40 Oct 30 19:11:04.001239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:03.999058 4789 scope.go:122] "RemoveContainer" containerID="4dd4ed65d104042708ffaca8910ab0a9eec5178d0f15c08014925bc8061e364b" Oct 30 19:11:04.048621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.048548 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" event={"ID":"bc769b1d-710b-4225-974e-699467d4b70c","Type":"ContainerStarted","Data":"5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40"} Oct 30 19:11:04.049611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.049584 4789 scope.go:122] "RemoveContainer" containerID="b9affdc8dd65cbbfc91678d893879d1e5d671140e4f19b2db5c32e9f8a9eeeaa" Oct 30 19:11:04.076559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.076537 4789 scope.go:122] "RemoveContainer" containerID="5e765ce212499a4243ca23624826ed58c2389bdd7e55ace14648d89cd66becce" Oct 30 19:11:04.089997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.089967 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:11:04.139602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.137473 4789 scope.go:122] "RemoveContainer" containerID="1174bb28454ee4404f5b85e1d060f20c798e278027babfcd849ed45fdfc1740b" Oct 30 19:11:04.170084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:04.170064 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:11:05.055606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:05.055573 4789 generic.go:356] "Generic (PLEG): container finished" podID="bc769b1d-710b-4225-974e-699467d4b70c" containerID="dceb9c390b8910b707c51cfdc60d933896dea0781a6a3aa0e3c6f5e18f2b2a88" exitCode=0 Oct 30 19:11:05.056028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:05.055651 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" event={"ID":"bc769b1d-710b-4225-974e-699467d4b70c","Type":"ContainerDied","Data":"dceb9c390b8910b707c51cfdc60d933896dea0781a6a3aa0e3c6f5e18f2b2a88"} Oct 30 19:11:06.208427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.208401 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:06.363098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.362997 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\" (UniqueName: \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\") pod \"bc769b1d-710b-4225-974e-699467d4b70c\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " Oct 30 19:11:06.363299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.363169 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\") pod \"bc769b1d-710b-4225-974e-699467d4b70c\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " Oct 30 19:11:06.363387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.363323 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\") pod \"bc769b1d-710b-4225-974e-699467d4b70c\" (UID: \"bc769b1d-710b-4225-974e-699467d4b70c\") " Oct 30 19:11:06.364026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.363991 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca" pod "bc769b1d-710b-4225-974e-699467d4b70c" (UID: "bc769b1d-710b-4225-974e-699467d4b70c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:06.365771 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.365740 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr" pod "bc769b1d-710b-4225-974e-699467d4b70c" (UID: "bc769b1d-710b-4225-974e-699467d4b70c"). InnerVolumeSpecName "kube-api-access-q2gwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:06.365979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.365955 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca" pod "bc769b1d-710b-4225-974e-699467d4b70c" (UID: "bc769b1d-710b-4225-974e-699467d4b70c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:06.465188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.465165 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:06.465188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.465187 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gwr\" (UniqueName: \"kubernetes.io/projected/bc769b1d-710b-4225-974e-699467d4b70c-kube-api-access-q2gwr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:06.465354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:06.465197 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc769b1d-710b-4225-974e-699467d4b70c-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:07.069014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.068977 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" event={"ID":"bc769b1d-710b-4225-974e-699467d4b70c","Type":"ContainerDied","Data":"5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40"} Oct 30 19:11:07.069231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.069035 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb74c9320af1317a93e4feef83c273c1cf42e1f55b1418b6ef47192dc872a40" Oct 30 19:11:07.069231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.068990 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns" Oct 30 19:11:07.577164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.577128 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm"] Oct 30 19:11:07.577642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.577627 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bc769b1d-710b-4225-974e-699467d4b70c" containerName="revoke-keys" Oct 30 19:11:07.577695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.577644 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc769b1d-710b-4225-974e-699467d4b70c" containerName="revoke-keys" Oct 30 19:11:07.577770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.577760 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc769b1d-710b-4225-974e-699467d4b70c" containerName="revoke-keys" Oct 30 19:11:07.581379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.581361 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.633274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.633232 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:07.633425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.633406 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:07.633804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.633783 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:07.637035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.637014 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm"] Oct 30 19:11:07.676270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.676246 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.676377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.676342 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpnw\" (UniqueName: \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.676459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.676445 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.679305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.679285 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:07.777706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.777678 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.777826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.777767 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.777826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.777805 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpnw\" (UniqueName: \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.778386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.778360 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.780144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.780118 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.836690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.836630 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpnw\" (UniqueName: \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\") pod \"maas-api-revoke-sub-2ed309e5-rpmwm\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:07.896345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:07.896324 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:08.274423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:08.274388 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm"] Oct 30 19:11:08.277676 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:08.277649 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e94802d_f491_4b13_97d2_df3bb5f50c86.slice/crio-11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a WatchSource:0}: Error finding container 11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a: Status 404 returned error can't find the container with id 11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a Oct 30 19:11:09.083752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:09.083723 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" event={"ID":"8e94802d-f491-4b13-97d2-df3bb5f50c86","Type":"ContainerStarted","Data":"d718528faf31bd33610e6307d2ff2c26bbf09606d43cd74fbfde90f67a6dd9c1"} Oct 30 19:11:09.084198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:09.083776 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" event={"ID":"8e94802d-f491-4b13-97d2-df3bb5f50c86","Type":"ContainerStarted","Data":"11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a"} Oct 30 19:11:09.197182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:09.197139 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" podStartSLOduration=2.197125827 podStartE2EDuration="2.197125827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:09.193582226 +0000 UTC m=+1505.695492715" watchObservedRunningTime="2026-10-30 19:11:09.197125827 +0000 UTC m=+1505.699036312" Oct 30 19:11:10.092778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:10.092746 4789 generic.go:356] "Generic (PLEG): container finished" podID="8e94802d-f491-4b13-97d2-df3bb5f50c86" containerID="d718528faf31bd33610e6307d2ff2c26bbf09606d43cd74fbfde90f67a6dd9c1" exitCode=0 Oct 30 19:11:10.093152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:10.092794 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" event={"ID":"8e94802d-f491-4b13-97d2-df3bb5f50c86","Type":"ContainerDied","Data":"d718528faf31bd33610e6307d2ff2c26bbf09606d43cd74fbfde90f67a6dd9c1"} Oct 30 19:11:11.258108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.258080 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:11.414614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.414470 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\") pod \"8e94802d-f491-4b13-97d2-df3bb5f50c86\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " Oct 30 19:11:11.414762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.414653 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\") pod \"8e94802d-f491-4b13-97d2-df3bb5f50c86\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " Oct 30 19:11:11.414841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.414810 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\" (UniqueName: \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\") pod \"8e94802d-f491-4b13-97d2-df3bb5f50c86\" (UID: \"8e94802d-f491-4b13-97d2-df3bb5f50c86\") " Oct 30 19:11:11.415480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.415426 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca" pod "8e94802d-f491-4b13-97d2-df3bb5f50c86" (UID: "8e94802d-f491-4b13-97d2-df3bb5f50c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:11.416786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.416757 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca" pod "8e94802d-f491-4b13-97d2-df3bb5f50c86" (UID: "8e94802d-f491-4b13-97d2-df3bb5f50c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:11.417378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.417349 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw" pod "8e94802d-f491-4b13-97d2-df3bb5f50c86" (UID: "8e94802d-f491-4b13-97d2-df3bb5f50c86"). InnerVolumeSpecName "kube-api-access-gbpnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:11.516089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.516053 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpnw\" (UniqueName: \"kubernetes.io/projected/8e94802d-f491-4b13-97d2-df3bb5f50c86-kube-api-access-gbpnw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:11.516089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.516090 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:11.516291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:11.516105 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e94802d-f491-4b13-97d2-df3bb5f50c86-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:12.106437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:12.106406 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" Oct 30 19:11:12.106437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:12.106420 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm" event={"ID":"8e94802d-f491-4b13-97d2-df3bb5f50c86","Type":"ContainerDied","Data":"11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a"} Oct 30 19:11:12.106644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:12.106480 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11319994c125d97dd6aa40ad1904b62b3b2d19825c97f4161f09b62511c2647a" Oct 30 19:11:13.318801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.318765 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz"] Oct 30 19:11:13.319465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.319443 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8e94802d-f491-4b13-97d2-df3bb5f50c86" containerName="revoke-keys" Oct 30 19:11:13.319465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.319464 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e94802d-f491-4b13-97d2-df3bb5f50c86" containerName="revoke-keys" Oct 30 19:11:13.319632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.319595 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e94802d-f491-4b13-97d2-df3bb5f50c86" containerName="revoke-keys" Oct 30 19:11:13.324607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.324586 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.379902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.379875 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz"] Oct 30 19:11:13.423071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.423039 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:13.423300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.423276 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:13.424690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.424666 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:13.424786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.424749 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:13.438118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.438088 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.438333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.438309 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxng\" (UniqueName: \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.438454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.438437 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.529454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.529428 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:11:13.533233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.533217 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.539176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.539157 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxng\" (UniqueName: \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.539290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.539256 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.539352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.539314 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.540311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.540290 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.541846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.541823 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.582303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.582247 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-1323a7-openshift-default-dockercfg-2p7xv\"" Oct 30 19:11:13.592419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.592397 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:11:13.598610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.598200 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxng\" (UniqueName: \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\") pod \"maas-api-revoke-sub-5b62a5dc-znnlz\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.640562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640528 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640601 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640631 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640687 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640724 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640778 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640805 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskcf\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.640881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640866 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.641078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.640901 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.641078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.641040 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.741935 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.741983 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742049 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742084 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742141 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742165 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskcf\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742243 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742276 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.742529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.742375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.743515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.743102 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.743515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.743397 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.744395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.744098 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.744395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.744154 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.744395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.744359 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.745402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.745360 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.746297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.745965 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.800699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.800662 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.801108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.801089 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskcf\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:13.858120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:13.858052 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:14.033042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.032937 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz"] Oct 30 19:11:14.037757 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:14.037726 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b66bde_fef1_4ce8_ab86_a42e2d36a48f.slice/crio-e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853 WatchSource:0}: Error finding container e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853: Status 404 returned error can't find the container with id e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853 Oct 30 19:11:14.125309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.125237 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" event={"ID":"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f","Type":"ContainerStarted","Data":"e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853"} Oct 30 19:11:14.244853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.244825 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:11:14.248723 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:14.248693 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a115530_4069_411e_9ab5_289553297b1b.slice/crio-1f6315a6bee528cbe6f9f98b175cdb48427375215a41480c2cf731207c59df62 WatchSource:0}: Error finding container 1f6315a6bee528cbe6f9f98b175cdb48427375215a41480c2cf731207c59df62: Status 404 returned error can't find the container with id 1f6315a6bee528cbe6f9f98b175cdb48427375215a41480c2cf731207c59df62 Oct 30 19:11:14.251116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.251081 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:14.251194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.251163 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:14.251262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:14.251205 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:15.136064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.136023 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" event={"ID":"4a115530-4069-411e-9ab5-289553297b1b","Type":"ContainerStarted","Data":"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec"} Oct 30 19:11:15.136593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.136085 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" event={"ID":"4a115530-4069-411e-9ab5-289553297b1b","Type":"ContainerStarted","Data":"1f6315a6bee528cbe6f9f98b175cdb48427375215a41480c2cf731207c59df62"} Oct 30 19:11:15.137856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.137829 4789 generic.go:356] "Generic (PLEG): container finished" podID="f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" containerID="787131e8bfbc92c1462346d3ccb8bd2c469f3751d0cf75b966deb663b5d43873" exitCode=0 Oct 30 19:11:15.137991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.137887 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" event={"ID":"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f","Type":"ContainerDied","Data":"787131e8bfbc92c1462346d3ccb8bd2c469f3751d0cf75b966deb663b5d43873"} Oct 30 19:11:15.253946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.253894 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podStartSLOduration=2.253879748 podStartE2EDuration="2.253879748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:15.251183001 +0000 UTC m=+1511.753093487" watchObservedRunningTime="2026-10-30 19:11:15.253879748 +0000 UTC m=+1511.755790236" Oct 30 19:11:15.858641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.858607 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:15.860433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.860406 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:15.860567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:15.860482 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:16.362557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.354451 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:11:16.362557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.359503 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.414872 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.414952 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415056 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415130 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgqv\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415181 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415257 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415313 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.463547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.415353 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523579 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523647 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523694 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523761 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgqv\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523809 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523852 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523900 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.523936 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.524019 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.524679 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.524992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.524977 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.528234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.525444 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.606232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.598570 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.653252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.650833 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.653252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.651444 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.782103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.777519 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:11:16.782103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.780465 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.782103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.781521 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.790804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.789455 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgqv\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:16.856144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.844199 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:16.860656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.860624 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:16.860938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.860893 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:16.974614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.974571 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\") pod \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " Oct 30 19:11:16.974887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.974865 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\" (UniqueName: \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\") pod \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " Oct 30 19:11:16.975804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.975781 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\") pod \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\" (UID: \"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f\") " Oct 30 19:11:16.981425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:16.981399 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca" pod "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" (UID: "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:17.055581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.055485 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca" pod "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" (UID: "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:17.055880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.055835 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng" pod "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" (UID: "f1b66bde-fef1-4ce8-ab86-a42e2d36a48f"). InnerVolumeSpecName "kube-api-access-4zxng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:17.078135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.078059 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:17.078135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.078092 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxng\" (UniqueName: \"kubernetes.io/projected/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-kube-api-access-4zxng\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:17.078135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.078108 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:17.101550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.081368 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:17.203283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.201857 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" Oct 30 19:11:17.203808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.203579 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz" event={"ID":"f1b66bde-fef1-4ce8-ab86-a42e2d36a48f","Type":"ContainerDied","Data":"e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853"} Oct 30 19:11:17.203808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.203631 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4a817255d182145a03a91e2ef9c85ca0dcb38e997cfaab8a88bce674bc3c853" Oct 30 19:11:17.763362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.763309 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:11:17.772633 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:17.772604 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8410804_6293_453e_8fb0_62e1fa943e68.slice/crio-96af1ec487fe644942705807f85072f945557007b89dadadc00c90a080119dbb WatchSource:0}: Error finding container 96af1ec487fe644942705807f85072f945557007b89dadadc00c90a080119dbb: Status 404 returned error can't find the container with id 96af1ec487fe644942705807f85072f945557007b89dadadc00c90a080119dbb Oct 30 19:11:17.777979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.776460 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:17.777979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.776544 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:17.777979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.776589 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:11:17.859397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.858966 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:17.859397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:17.859023 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:18.218389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.218350 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" event={"ID":"d8410804-6293-453e-8fb0-62e1fa943e68","Type":"ContainerStarted","Data":"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6"} Oct 30 19:11:18.218653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.218626 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" event={"ID":"d8410804-6293-453e-8fb0-62e1fa943e68","Type":"ContainerStarted","Data":"96af1ec487fe644942705807f85072f945557007b89dadadc00c90a080119dbb"} Oct 30 19:11:18.289441 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:11:18.289405 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:11:18.340248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.335261 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podStartSLOduration=2.335242093 podStartE2EDuration="2.335242093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:18.334739847 +0000 UTC m=+1514.836650334" watchObservedRunningTime="2026-10-30 19:11:18.335242093 +0000 UTC m=+1514.837152580" Oct 30 19:11:18.498200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.498117 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld"] Oct 30 19:11:18.499203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.499174 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" containerName="revoke-keys" Oct 30 19:11:18.504230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.499364 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" containerName="revoke-keys" Oct 30 19:11:18.504531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.504510 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" containerName="revoke-keys" Oct 30 19:11:18.517461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.517442 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.553527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.552044 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld"] Oct 30 19:11:18.572256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.572145 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:18.572256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.572196 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:18.577132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.577097 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ql6n\" (UniqueName: \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.577597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.577569 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.577803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.577780 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.587806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.587787 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:18.587943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.587843 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:18.683861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.683824 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ql6n\" (UniqueName: \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.684438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.684410 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.684674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.684652 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.685093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.685071 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.686998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.686980 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.746905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.746867 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ql6n\" (UniqueName: \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\") pod \"maas-api-revoke-sub-3dc44446-pcgld\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.840907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.840818 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:18.860691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.859710 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:18.860691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:18.859792 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:19.114784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.114286 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:19.114784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.114425 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:19.114784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.114471 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:19.341604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.341549 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld"] Oct 30 19:11:19.859365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.859339 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:19.859685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:19.859390 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:20.090201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.090116 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:20.090201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.090166 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:20.251229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.251172 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" event={"ID":"e78123e3-c274-49d4-aa21-0a724b74ed75","Type":"ContainerStarted","Data":"f882b83731e7b5da02eaa0282ee28e5b9397985cd1f5ee8021b344b75da0a266"} Oct 30 19:11:20.251395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.251245 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" event={"ID":"e78123e3-c274-49d4-aa21-0a724b74ed75","Type":"ContainerStarted","Data":"c6ff6fbc536be3e252ff272c9d051e58c83d1ce1f5544dd6f22fd82ef2d15e19"} Oct 30 19:11:20.365316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.365201 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" podStartSLOduration=2.365187558 podStartE2EDuration="2.365187558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:20.363743322 +0000 UTC m=+1516.865653833" watchObservedRunningTime="2026-10-30 19:11:20.365187558 +0000 UTC m=+1516.867098043" Oct 30 19:11:20.858738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.858701 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 19:11:20.858909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:20.858758 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 19:11:21.090531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:21.090484 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:21.090994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:21.090564 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:21.259319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:21.259284 4789 generic.go:356] "Generic (PLEG): container finished" podID="e78123e3-c274-49d4-aa21-0a724b74ed75" containerID="f882b83731e7b5da02eaa0282ee28e5b9397985cd1f5ee8021b344b75da0a266" exitCode=0 Oct 30 19:11:21.259514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:21.259365 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" event={"ID":"e78123e3-c274-49d4-aa21-0a724b74ed75","Type":"ContainerDied","Data":"f882b83731e7b5da02eaa0282ee28e5b9397985cd1f5ee8021b344b75da0a266"} Oct 30 19:11:22.043845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.043809 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:22.044277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.044248 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:22.045004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.044984 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:11:22.090731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.090704 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:22.091072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.090755 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:22.531414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.531389 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:22.578728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.578700 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\" (UniqueName: \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\") pod \"e78123e3-c274-49d4-aa21-0a724b74ed75\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " Oct 30 19:11:22.578870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.578784 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\") pod \"e78123e3-c274-49d4-aa21-0a724b74ed75\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " Oct 30 19:11:22.578870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.578837 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\") pod \"e78123e3-c274-49d4-aa21-0a724b74ed75\" (UID: \"e78123e3-c274-49d4-aa21-0a724b74ed75\") " Oct 30 19:11:22.579153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.579131 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca" pod "e78123e3-c274-49d4-aa21-0a724b74ed75" (UID: "e78123e3-c274-49d4-aa21-0a724b74ed75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:22.579332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.579316 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:22.580902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.580873 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n" pod "e78123e3-c274-49d4-aa21-0a724b74ed75" (UID: "e78123e3-c274-49d4-aa21-0a724b74ed75"). InnerVolumeSpecName "kube-api-access-8ql6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:22.581082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.581059 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca" pod "e78123e3-c274-49d4-aa21-0a724b74ed75" (UID: "e78123e3-c274-49d4-aa21-0a724b74ed75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:22.680309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.680241 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ql6n\" (UniqueName: \"kubernetes.io/projected/e78123e3-c274-49d4-aa21-0a724b74ed75-kube-api-access-8ql6n\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:22.680309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:22.680268 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78123e3-c274-49d4-aa21-0a724b74ed75-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:23.091085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:23.091052 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:23.091607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:23.091117 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:23.274482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:23.274454 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" Oct 30 19:11:23.274623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:23.274493 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld" event={"ID":"e78123e3-c274-49d4-aa21-0a724b74ed75","Type":"ContainerDied","Data":"c6ff6fbc536be3e252ff272c9d051e58c83d1ce1f5544dd6f22fd82ef2d15e19"} Oct 30 19:11:23.274623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:23.274540 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ff6fbc536be3e252ff272c9d051e58c83d1ce1f5544dd6f22fd82ef2d15e19" Oct 30 19:11:24.090811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.090778 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:11:24.091026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.090835 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:11:24.153170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.153136 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn"] Oct 30 19:11:24.153781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.153754 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e78123e3-c274-49d4-aa21-0a724b74ed75" containerName="revoke-keys" Oct 30 19:11:24.153781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.153775 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78123e3-c274-49d4-aa21-0a724b74ed75" containerName="revoke-keys" Oct 30 19:11:24.153939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.153894 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78123e3-c274-49d4-aa21-0a724b74ed75" containerName="revoke-keys" Oct 30 19:11:24.157546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.157529 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.208127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.208095 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:24.208127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.208105 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:24.208375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.208104 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:24.208375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.208162 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:24.222824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.222798 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.222973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.222835 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.222973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.222857 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h545\" (UniqueName: \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.259531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.259497 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn"] Oct 30 19:11:24.323638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.323610 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.323810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.323648 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.323810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.323673 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h545\" (UniqueName: \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.324322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.324301 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.326262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.326236 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.382430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.382362 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h545\" (UniqueName: \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\") pod \"maas-api-revoke-sub-0dc1a9fd-n5fzn\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.475944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.475912 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:24.856739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:24.856704 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn"] Oct 30 19:11:24.860908 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:24.860869 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0158847_6a0a_4604_87da_a1c015e425ee.slice/crio-80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a WatchSource:0}: Error finding container 80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a: Status 404 returned error can't find the container with id 80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a Oct 30 19:11:25.289082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.289048 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" event={"ID":"a0158847-6a0a-4604-87da-a1c015e425ee","Type":"ContainerStarted","Data":"c895baba160fc3c239fa9ef42302786247d70b64ba33f5aed8ebc118df483870"} Oct 30 19:11:25.289448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.289110 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" event={"ID":"a0158847-6a0a-4604-87da-a1c015e425ee","Type":"ContainerStarted","Data":"80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a"} Oct 30 19:11:25.403181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.403070 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" podStartSLOduration=1.403053672 podStartE2EDuration="1.403053672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:25.398782138 +0000 UTC m=+1521.900692627" watchObservedRunningTime="2026-10-30 19:11:25.403053672 +0000 UTC m=+1521.904964159" Oct 30 19:11:25.450448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.450420 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:25.450960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.450934 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:25.451685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:25.451653 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:11:26.297134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:26.297104 4789 generic.go:356] "Generic (PLEG): container finished" podID="a0158847-6a0a-4604-87da-a1c015e425ee" containerID="c895baba160fc3c239fa9ef42302786247d70b64ba33f5aed8ebc118df483870" exitCode=0 Oct 30 19:11:26.297618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:26.297183 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" event={"ID":"a0158847-6a0a-4604-87da-a1c015e425ee","Type":"ContainerDied","Data":"c895baba160fc3c239fa9ef42302786247d70b64ba33f5aed8ebc118df483870"} Oct 30 19:11:27.493395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.493365 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:27.563614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.563543 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\") pod \"a0158847-6a0a-4604-87da-a1c015e425ee\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " Oct 30 19:11:27.563760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.563659 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\" (UniqueName: \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\") pod \"a0158847-6a0a-4604-87da-a1c015e425ee\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " Oct 30 19:11:27.563870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.563836 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\") pod \"a0158847-6a0a-4604-87da-a1c015e425ee\" (UID: \"a0158847-6a0a-4604-87da-a1c015e425ee\") " Oct 30 19:11:27.563870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.563847 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca" pod "a0158847-6a0a-4604-87da-a1c015e425ee" (UID: "a0158847-6a0a-4604-87da-a1c015e425ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:27.564202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.564185 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:27.565752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.565729 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545" pod "a0158847-6a0a-4604-87da-a1c015e425ee" (UID: "a0158847-6a0a-4604-87da-a1c015e425ee"). InnerVolumeSpecName "kube-api-access-8h545". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:27.565823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.565801 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca" pod "a0158847-6a0a-4604-87da-a1c015e425ee" (UID: "a0158847-6a0a-4604-87da-a1c015e425ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:27.665755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.665725 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0158847-6a0a-4604-87da-a1c015e425ee-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:27.665755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:27.665755 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h545\" (UniqueName: \"kubernetes.io/projected/a0158847-6a0a-4604-87da-a1c015e425ee-kube-api-access-8h545\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:28.311457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:28.311424 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" Oct 30 19:11:28.311629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:28.311467 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn" event={"ID":"a0158847-6a0a-4604-87da-a1c015e425ee","Type":"ContainerDied","Data":"80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a"} Oct 30 19:11:28.311629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:28.311517 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d3b57444a7de1250b0910c4a662eb0c87c2187d578becb0f02392084d7061a" Oct 30 19:11:35.304413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.304371 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m"] Oct 30 19:11:35.315538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.315497 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a0158847-6a0a-4604-87da-a1c015e425ee" containerName="revoke-keys" Oct 30 19:11:35.315758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.315742 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0158847-6a0a-4604-87da-a1c015e425ee" containerName="revoke-keys" Oct 30 19:11:35.316092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.316079 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0158847-6a0a-4604-87da-a1c015e425ee" containerName="revoke-keys" Oct 30 19:11:35.322423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.322399 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:35.394296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.394263 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m"] Oct 30 19:11:35.444254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.420267 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:35.444524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.440408 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:35.444632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.440490 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:35.735288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.440583 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:35.784422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.784372 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:35.785321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.785288 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:11:35.874255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.841961 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 19:11:35.874255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:35.843018 4789 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:36.265413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.265372 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:36.472234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.397702 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 30 19:11:36.529284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.455025 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:36.578371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.578322 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:11:36.641234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.595506 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xw6\" (UniqueName: \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.641234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.614459 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.641234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.614681 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.747232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.722519 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.782076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.781999 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.793912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.793885 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xw6\" (UniqueName: \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.818940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.723968 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.829602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.793301 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:11:36.857230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.750795 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2"] Oct 30 19:11:36.859256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.856534 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:36.873268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.873240 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:36.895471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.895440 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:11:36.898396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.898377 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2"] Oct 30 19:11:36.898592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.898577 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:11:36.899556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.899538 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:36.967326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:36.960798 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xw6\" (UniqueName: \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\") pod \"maas-api-revoke-sub-e2baf277-8sp4m\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:37.037177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.029085 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.037177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.029239 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.037177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.029332 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4hb\" (UniqueName: \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.051683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.051611 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:11:37.097196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.097158 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:37.098951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.098902 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 19:11:37.136822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.136786 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.138364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.137203 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.138639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.138618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.149847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.149821 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4hb\" (UniqueName: \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.189131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.188980 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:37.190585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.190559 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.280584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.280555 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4hb\" (UniqueName: \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\") pod \"maas-api-revoke-sub-6f717d9e-mxgx2\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.381777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.381407 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:37.626479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.622108 4789 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Oct 30 19:11:37.650234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.648356 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Oct 30 19:11:37.650234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.648771 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:11:37.686230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.652326 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:11:37.686230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.652341 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:11:37.686230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.652666 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:11:37.686230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.652720 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:11:37.881656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.881568 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:11:37.906233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.902455 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:11:37.955975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:37.955946 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:11:38.178339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.177761 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m"] Oct 30 19:11:38.183552 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:38.183523 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90349ec0_fada_4176_82e1_c09b23f3d2cb.slice/crio-99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96 WatchSource:0}: Error finding container 99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96: Status 404 returned error can't find the container with id 99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96 Oct 30 19:11:38.357127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.357097 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2"] Oct 30 19:11:38.360927 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:38.360899 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179a0856_b1df_4c33_83e1_d25c3422f334.slice/crio-ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d WatchSource:0}: Error finding container ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d: Status 404 returned error can't find the container with id ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d Oct 30 19:11:38.742467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.742439 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" event={"ID":"179a0856-b1df-4c33-83e1-d25c3422f334","Type":"ContainerStarted","Data":"3527c255f687537789601019a284f9c6c7e4da4228166fa5555a0a0ea66d079c"} Oct 30 19:11:38.742800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.742494 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" event={"ID":"179a0856-b1df-4c33-83e1-d25c3422f334","Type":"ContainerStarted","Data":"ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d"} Oct 30 19:11:38.744159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.744127 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" event={"ID":"90349ec0-fada-4176-82e1-c09b23f3d2cb","Type":"ContainerStarted","Data":"aaa04bb9f4bc3ebe4c37aea7f8aee54e9afd8527f7d20d4439bc03e5b83de672"} Oct 30 19:11:38.744290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.744179 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" event={"ID":"90349ec0-fada-4176-82e1-c09b23f3d2cb","Type":"ContainerStarted","Data":"99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96"} Oct 30 19:11:38.858814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.858718 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" podStartSLOduration=3.858704478 podStartE2EDuration="3.858704478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:38.85486782 +0000 UTC m=+1535.356778315" watchObservedRunningTime="2026-10-30 19:11:38.858704478 +0000 UTC m=+1535.360614964" Oct 30 19:11:38.967602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:38.967554 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" podStartSLOduration=3.967540923 podStartE2EDuration="3.967540923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:38.966198047 +0000 UTC m=+1535.468108544" watchObservedRunningTime="2026-10-30 19:11:38.967540923 +0000 UTC m=+1535.469451408" Oct 30 19:11:39.752292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:39.752254 4789 generic.go:356] "Generic (PLEG): container finished" podID="179a0856-b1df-4c33-83e1-d25c3422f334" containerID="3527c255f687537789601019a284f9c6c7e4da4228166fa5555a0a0ea66d079c" exitCode=0 Oct 30 19:11:39.752751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:39.752340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" event={"ID":"179a0856-b1df-4c33-83e1-d25c3422f334","Type":"ContainerDied","Data":"3527c255f687537789601019a284f9c6c7e4da4228166fa5555a0a0ea66d079c"} Oct 30 19:11:39.753951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:39.753927 4789 generic.go:356] "Generic (PLEG): container finished" podID="90349ec0-fada-4176-82e1-c09b23f3d2cb" containerID="aaa04bb9f4bc3ebe4c37aea7f8aee54e9afd8527f7d20d4439bc03e5b83de672" exitCode=0 Oct 30 19:11:39.754076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:39.754005 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" event={"ID":"90349ec0-fada-4176-82e1-c09b23f3d2cb","Type":"ContainerDied","Data":"aaa04bb9f4bc3ebe4c37aea7f8aee54e9afd8527f7d20d4439bc03e5b83de672"} Oct 30 19:11:40.928516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:40.928494 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:40.963621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:40.963594 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:41.021143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021076 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\") pod \"179a0856-b1df-4c33-83e1-d25c3422f334\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " Oct 30 19:11:41.021293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021154 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\" (UniqueName: \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\") pod \"90349ec0-fada-4176-82e1-c09b23f3d2cb\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " Oct 30 19:11:41.021293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021203 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\") pod \"179a0856-b1df-4c33-83e1-d25c3422f334\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " Oct 30 19:11:41.021385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021318 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\" (UniqueName: \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\") pod \"179a0856-b1df-4c33-83e1-d25c3422f334\" (UID: \"179a0856-b1df-4c33-83e1-d25c3422f334\") " Oct 30 19:11:41.021385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021364 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\") pod \"90349ec0-fada-4176-82e1-c09b23f3d2cb\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " Oct 30 19:11:41.021530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021512 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\") pod \"90349ec0-fada-4176-82e1-c09b23f3d2cb\" (UID: \"90349ec0-fada-4176-82e1-c09b23f3d2cb\") " Oct 30 19:11:41.021975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021649 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca" pod "179a0856-b1df-4c33-83e1-d25c3422f334" (UID: "179a0856-b1df-4c33-83e1-d25c3422f334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:41.021975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021936 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca" pod "90349ec0-fada-4176-82e1-c09b23f3d2cb" (UID: "90349ec0-fada-4176-82e1-c09b23f3d2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:41.021975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.021966 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.023470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.023444 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca" pod "90349ec0-fada-4176-82e1-c09b23f3d2cb" (UID: "90349ec0-fada-4176-82e1-c09b23f3d2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:41.023556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.023510 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb" pod "179a0856-b1df-4c33-83e1-d25c3422f334" (UID: "179a0856-b1df-4c33-83e1-d25c3422f334"). InnerVolumeSpecName "kube-api-access-jv4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:41.023645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.023626 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca" pod "179a0856-b1df-4c33-83e1-d25c3422f334" (UID: "179a0856-b1df-4c33-83e1-d25c3422f334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:41.023645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.023637 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6" pod "90349ec0-fada-4176-82e1-c09b23f3d2cb" (UID: "90349ec0-fada-4176-82e1-c09b23f3d2cb"). InnerVolumeSpecName "kube-api-access-n9xw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:41.123365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.123336 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv4hb\" (UniqueName: \"kubernetes.io/projected/179a0856-b1df-4c33-83e1-d25c3422f334-kube-api-access-jv4hb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.123365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.123362 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.123365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.123372 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90349ec0-fada-4176-82e1-c09b23f3d2cb-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.123568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.123383 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/179a0856-b1df-4c33-83e1-d25c3422f334-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.123568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.123392 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9xw6\" (UniqueName: \"kubernetes.io/projected/90349ec0-fada-4176-82e1-c09b23f3d2cb-kube-api-access-n9xw6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:41.781199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.781167 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" Oct 30 19:11:41.781199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.781200 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2" event={"ID":"179a0856-b1df-4c33-83e1-d25c3422f334","Type":"ContainerDied","Data":"ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d"} Oct 30 19:11:41.781477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.781291 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebb0d66612fad0a35a0e7148f325386b33653001d511df4b9a94a411a323f33d" Oct 30 19:11:41.782615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.782592 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" Oct 30 19:11:41.782711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.782616 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m" event={"ID":"90349ec0-fada-4176-82e1-c09b23f3d2cb","Type":"ContainerDied","Data":"99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96"} Oct 30 19:11:41.782711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:41.782663 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99a57300b4c95921b1f7a661a5db5d33575678ad88f048243b4d080def1bdb96" Oct 30 19:11:42.223772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.223738 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:11:42.224274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224259 4789 cpu_manager.go:414] "containerMap: removing container" podUID="179a0856-b1df-4c33-83e1-d25c3422f334" containerName="revoke-keys" Oct 30 19:11:42.224324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224274 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179a0856-b1df-4c33-83e1-d25c3422f334" containerName="revoke-keys" Oct 30 19:11:42.224324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224291 4789 cpu_manager.go:414] "containerMap: removing container" podUID="90349ec0-fada-4176-82e1-c09b23f3d2cb" containerName="revoke-keys" Oct 30 19:11:42.224396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224299 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90349ec0-fada-4176-82e1-c09b23f3d2cb" containerName="revoke-keys" Oct 30 19:11:42.224545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224531 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="179a0856-b1df-4c33-83e1-d25c3422f334" containerName="revoke-keys" Oct 30 19:11:42.224581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.224551 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="90349ec0-fada-4176-82e1-c09b23f3d2cb" containerName="revoke-keys" Oct 30 19:11:42.228004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.227985 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.278103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.278075 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-942c5e85-dockercfg-44ksk\"" Oct 30 19:11:42.278913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.278895 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-942c5e85\"" Oct 30 19:11:42.287579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.287547 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:11:42.338265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.338238 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb8vv\" (UniqueName: \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.338456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.338437 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.445865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.445828 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.446065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.445918 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb8vv\" (UniqueName: \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.446975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.446947 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.503917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.503828 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb8vv\" (UniqueName: \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\") pod \"payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.541475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.541438 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:42.927984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:42.927897 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:11:42.932302 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:42.932274 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33a3ba1_08f1_486c_99a6_bfb48a8f6732.slice/crio-d9800aea0689affc27754b1ecb57ffb591c31bbd3cb95e8c1e243d50181d9046 WatchSource:0}: Error finding container d9800aea0689affc27754b1ecb57ffb591c31bbd3cb95e8c1e243d50181d9046: Status 404 returned error can't find the container with id d9800aea0689affc27754b1ecb57ffb591c31bbd3cb95e8c1e243d50181d9046 Oct 30 19:11:43.270636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.270609 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz"] Oct 30 19:11:43.274656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.274637 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.327394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.327363 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:43.327592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.327570 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:43.327663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.327583 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:43.327663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.327609 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:43.356446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.356416 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvg4\" (UniqueName: \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.356618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.356468 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.356618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.356565 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.357406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.357382 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz"] Oct 30 19:11:43.458134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.458104 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.458305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.458247 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvg4\" (UniqueName: \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.458305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.458276 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.458859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.458838 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.460370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.460350 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.517857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.517825 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvg4\" (UniqueName: \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\") pod \"maas-api-revoke-sub-65e42482-jjmlz\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.588372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.588312 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:43.797549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.797513 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" event={"ID":"b33a3ba1-08f1-486c-99a6-bfb48a8f6732","Type":"ContainerStarted","Data":"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a"} Oct 30 19:11:43.797737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.797577 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" event={"ID":"b33a3ba1-08f1-486c-99a6-bfb48a8f6732","Type":"ContainerStarted","Data":"d9800aea0689affc27754b1ecb57ffb591c31bbd3cb95e8c1e243d50181d9046"} Oct 30 19:11:43.797737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.797612 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:43.914270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:43.914162 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" podStartSLOduration=1.914149406 podStartE2EDuration="1.914149406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:43.907302297 +0000 UTC m=+1540.409212783" watchObservedRunningTime="2026-10-30 19:11:43.914149406 +0000 UTC m=+1540.416059902" Oct 30 19:11:44.032099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:44.032065 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz"] Oct 30 19:11:44.037646 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:44.037616 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccb8991_e573_41c6_9b4b_f0f7aa51a0e9.slice/crio-07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422 WatchSource:0}: Error finding container 07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422: Status 404 returned error can't find the container with id 07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422 Oct 30 19:11:44.806752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:44.806710 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" event={"ID":"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9","Type":"ContainerStarted","Data":"7706db4d8c688bced6cbbdc7e331ead961c90e45176f82550abe57450c95b04c"} Oct 30 19:11:44.807252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:44.806805 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" event={"ID":"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9","Type":"ContainerStarted","Data":"07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422"} Oct 30 19:11:44.923221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:44.923157 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" podStartSLOduration=1.9231434219999999 podStartE2EDuration="1.923143422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:44.91779964 +0000 UTC m=+1541.419710126" watchObservedRunningTime="2026-10-30 19:11:44.923143422 +0000 UTC m=+1541.425053907" Oct 30 19:11:45.817382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:45.817342 4789 generic.go:356] "Generic (PLEG): container finished" podID="8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" containerID="7706db4d8c688bced6cbbdc7e331ead961c90e45176f82550abe57450c95b04c" exitCode=0 Oct 30 19:11:45.817795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:45.817398 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" event={"ID":"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9","Type":"ContainerDied","Data":"7706db4d8c688bced6cbbdc7e331ead961c90e45176f82550abe57450c95b04c"} Oct 30 19:11:46.971431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:46.971406 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:47.105195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.105117 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\") pod \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " Oct 30 19:11:47.105195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.105189 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\") pod \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " Oct 30 19:11:47.105408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.105275 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\" (UniqueName: \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\") pod \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\" (UID: \"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9\") " Oct 30 19:11:47.105713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.105680 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca" pod "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" (UID: "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:47.105846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.105833 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:47.107512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.107483 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4" pod "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" (UID: "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9"). InnerVolumeSpecName "kube-api-access-5mvg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:47.107798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.107771 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca" pod "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" (UID: "8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:47.207295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.207269 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:47.207295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.207293 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvg4\" (UniqueName: \"kubernetes.io/projected/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9-kube-api-access-5mvg4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:47.832374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.832345 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" Oct 30 19:11:47.832530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.832366 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz" event={"ID":"8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9","Type":"ContainerDied","Data":"07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422"} Oct 30 19:11:47.832530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:47.832419 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e43962d7141ead56a69f9050d62567c0f0c7cebb61777bc154ad43c60bd422" Oct 30 19:11:48.196063 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:11:48.196028 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:11:49.764084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.764046 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5"] Oct 30 19:11:49.764761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.764738 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" containerName="revoke-keys" Oct 30 19:11:49.764761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.764762 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" containerName="revoke-keys" Oct 30 19:11:49.764965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.764951 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" containerName="revoke-keys" Oct 30 19:11:49.770269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.770205 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:49.821354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.821320 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:11:49.821550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.821392 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:11:49.867739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.867692 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:49.867739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.867734 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:11:49.871443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.871407 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5"] Oct 30 19:11:49.935520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.935493 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:49.935707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.935688 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:49.935757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:49.935719 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8grn\" (UniqueName: \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.036876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.036809 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.036876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.036842 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8grn\" (UniqueName: \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.037028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.036899 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.037475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.037453 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.039202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.039183 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.095258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.095237 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8grn\" (UniqueName: \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\") pod \"maas-api-revoke-sub-19efe8a8-98lt5\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.391914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.391845 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:50.767777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.767743 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5"] Oct 30 19:11:50.772024 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:11:50.771996 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b89a7f8_7870_40b3_a63c_f90aa5e9aaf6.slice/crio-1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881 WatchSource:0}: Error finding container 1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881: Status 404 returned error can't find the container with id 1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881 Oct 30 19:11:50.853185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:50.853156 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" event={"ID":"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6","Type":"ContainerStarted","Data":"1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881"} Oct 30 19:11:51.861438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:51.861406 4789 generic.go:356] "Generic (PLEG): container finished" podID="8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" containerID="44d45e4ce3940297391a38799dac7ebfa1334f7de6fbad9ffb6144d2d7b72d7b" exitCode=0 Oct 30 19:11:51.861831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:51.861442 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" event={"ID":"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6","Type":"ContainerDied","Data":"44d45e4ce3940297391a38799dac7ebfa1334f7de6fbad9ffb6144d2d7b72d7b"} Oct 30 19:11:53.023618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.023594 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:53.170864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.170781 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\" (UniqueName: \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\") pod \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " Oct 30 19:11:53.171023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.170895 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\") pod \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " Oct 30 19:11:53.171023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.170978 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\") pod \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\" (UID: \"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6\") " Oct 30 19:11:53.171386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.171357 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca" pod "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" (UID: "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:53.171518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.171493 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:53.173245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.173134 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn" pod "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" (UID: "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6"). InnerVolumeSpecName "kube-api-access-m8grn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:53.173549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.173520 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca" pod "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" (UID: "8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:53.272939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.272917 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8grn\" (UniqueName: \"kubernetes.io/projected/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-kube-api-access-m8grn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:53.272939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.272939 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:11:53.875704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.875680 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" Oct 30 19:11:53.875828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.875682 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5" event={"ID":"8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6","Type":"ContainerDied","Data":"1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881"} Oct 30 19:11:53.875828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:53.875800 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7d8299acbac22a8204de6f707a9209173e7f43088dcd8ef382659f18c73881" Oct 30 19:11:54.809773 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:54.809743 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:11:58.954408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:58.954367 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:11:58.955040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:58.954919 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" containerName="revoke-keys" Oct 30 19:11:58.955040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:58.954932 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" containerName="revoke-keys" Oct 30 19:11:58.955040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:58.955018 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" containerName="revoke-keys" Oct 30 19:11:58.959347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:58.959325 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.035744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.035707 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:11:59.137570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137530 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.137826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137613 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.137826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137640 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.137826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137725 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.137826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137751 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.141014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.137825 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.143107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.143057 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.143284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.143116 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stk8\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.143384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.143355 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244327 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244408 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244465 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244489 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244550 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244575 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244634 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244663 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.244910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.244690 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stk8\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.245062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.245035 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.245126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.245059 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.245362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.245339 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.245432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.245366 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.245432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.245399 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.247352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.247330 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.247826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.247803 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.302791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.302755 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.303061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.303021 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stk8\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:11:59.577475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:11:59.577384 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:12:00.011527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.011478 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:12:00.016888 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:12:00.016862 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5f4586_0772_4c45_8ba7_68cb4585ad78.slice/crio-ae53fb43950da1d6fde0484bbbf074cbe54f74fa7d9e505d62eb87696e46248d WatchSource:0}: Error finding container ae53fb43950da1d6fde0484bbbf074cbe54f74fa7d9e505d62eb87696e46248d: Status 404 returned error can't find the container with id ae53fb43950da1d6fde0484bbbf074cbe54f74fa7d9e505d62eb87696e46248d Oct 30 19:12:00.019100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.019066 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:00.019169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.019135 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:00.019169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.019163 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:00.491493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.485699 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:12:00.491493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.491291 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.537353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.537321 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:12:00.541537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.541515 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.563671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563643 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.563817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563680 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz85d\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.563817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563775 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.563944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563818 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.563944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563839 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.563944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.563915 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g5b\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564048 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564077 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564105 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564144 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564272 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564292 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564323 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564349 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564468 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.564560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564508 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564547 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.564794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.564641 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.585950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.585924 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:12:00.635315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.635274 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:12:00.666282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666241 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666306 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666352 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666392 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666449 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666478 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz85d\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666538 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666587 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666681 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g5b\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666711 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666733 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666735 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666762 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666788 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666849 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666873 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666925 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.666955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.666950 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.667531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.667432 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.667678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.667654 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.667742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.667692 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.667914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.667892 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.668541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.668514 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.668782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.668760 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.669004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.668982 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.669250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.669232 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.669827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.669802 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.671776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.670293 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.671776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.671431 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.672107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.672086 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.672811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.672779 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.729900 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.729202 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.732337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.730539 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g5b\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.737859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.737830 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.738713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.738691 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz85d\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.817282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.816428 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:00.856911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.856885 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:00.938251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.937836 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" event={"ID":"4f5f4586-0772-4c45-8ba7-68cb4585ad78","Type":"ContainerStarted","Data":"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e"} Oct 30 19:12:00.938251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:00.937890 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" event={"ID":"4f5f4586-0772-4c45-8ba7-68cb4585ad78","Type":"ContainerStarted","Data":"ae53fb43950da1d6fde0484bbbf074cbe54f74fa7d9e505d62eb87696e46248d"} Oct 30 19:12:01.061698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.061012 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podStartSLOduration=3.060998583 podStartE2EDuration="3.060998583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:01.056570663 +0000 UTC m=+1557.558481160" watchObservedRunningTime="2026-10-30 19:12:01.060998583 +0000 UTC m=+1557.562909068" Oct 30 19:12:01.258716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.258675 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:12:01.324192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.324164 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:12:01.324678 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:12:01.324649 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c5c660_734c_4584_82aa_7c3a8e6892e4.slice/crio-76abd3bc4931996f611f2146cbbaad6b5cdab7ea835f65a49482cdddbb477e9c WatchSource:0}: Error finding container 76abd3bc4931996f611f2146cbbaad6b5cdab7ea835f65a49482cdddbb477e9c: Status 404 returned error can't find the container with id 76abd3bc4931996f611f2146cbbaad6b5cdab7ea835f65a49482cdddbb477e9c Oct 30 19:12:01.326619 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:12:01.326571 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0024db27_849f_48c6_bed2_3a60fa7cba15.slice/crio-bf6e56158eb7bf056a31f5ee10d4891cf100a929ca4cf475113ca98da44e049e WatchSource:0}: Error finding container bf6e56158eb7bf056a31f5ee10d4891cf100a929ca4cf475113ca98da44e049e: Status 404 returned error can't find the container with id bf6e56158eb7bf056a31f5ee10d4891cf100a929ca4cf475113ca98da44e049e Oct 30 19:12:01.327731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.327700 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.327808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.327793 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.327859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.327840 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.329146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.329118 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.329286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.329171 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.329286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.329200 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:01.578321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.578020 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:12:01.579337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.579233 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:01.579337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.579280 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:01.947911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.947872 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" event={"ID":"47c5c660-734c-4584-82aa-7c3a8e6892e4","Type":"ContainerStarted","Data":"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f"} Oct 30 19:12:01.948097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.947945 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" event={"ID":"47c5c660-734c-4584-82aa-7c3a8e6892e4","Type":"ContainerStarted","Data":"76abd3bc4931996f611f2146cbbaad6b5cdab7ea835f65a49482cdddbb477e9c"} Oct 30 19:12:01.949744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.949713 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" event={"ID":"0024db27-849f-48c6-bed2-3a60fa7cba15","Type":"ContainerStarted","Data":"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af"} Oct 30 19:12:01.949881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:01.949772 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" event={"ID":"0024db27-849f-48c6-bed2-3a60fa7cba15","Type":"ContainerStarted","Data":"bf6e56158eb7bf056a31f5ee10d4891cf100a929ca4cf475113ca98da44e049e"} Oct 30 19:12:02.072461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.072413 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podStartSLOduration=2.072399835 podStartE2EDuration="2.072399835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:02.071181977 +0000 UTC m=+1558.573092463" watchObservedRunningTime="2026-10-30 19:12:02.072399835 +0000 UTC m=+1558.574310321" Oct 30 19:12:02.221303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.220626 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podStartSLOduration=2.220605256 podStartE2EDuration="2.220605256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:02.217761762 +0000 UTC m=+1558.719672250" watchObservedRunningTime="2026-10-30 19:12:02.220605256 +0000 UTC m=+1558.722515745" Oct 30 19:12:02.593295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.592758 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:02.593295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.592825 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:02.684234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.683631 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:12:02.734308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.730597 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.867766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.862271 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:02.867766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.864023 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:02.882372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.882327 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:02.882535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.882414 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:02.887239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.883376 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:02.887483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.887450 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:02.911395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.905422 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4f4614-openshift-default-dockercfg-h7frg\"" Oct 30 19:12:02.924480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.914063 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981050 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981146 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981194 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981294 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981326 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcjr\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981367 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981405 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981438 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:02.996236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:02.981478 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103718 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103788 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103873 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103911 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcjr\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103957 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.103996 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.104028 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.104067 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.104183 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.104676 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.105351 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.107252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.105575 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.133337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.127053 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.133337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.127357 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.218233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.212793 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.246230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.243764 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.246230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.244160 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.271235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.259130 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcjr\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\") pod \"e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.543243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.537392 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:03.581234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.580646 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:03.581234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.580710 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:03.819679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.819574 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:03.819949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.819903 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:03.868545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.865689 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:03.868545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.865743 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:03.910248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:03.904298 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:04.168381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.168289 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:04.342812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.342775 4789 scope.go:122] "RemoveContainer" containerID="dbe2602f777de0e18f34394201751bd077068f37f1fb7e625383457c26bf1e00" Oct 30 19:12:04.390311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.388705 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:12:04.506684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.506643 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:04.506894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.506874 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:04.507030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.507013 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:04.566672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.566648 4789 scope.go:122] "RemoveContainer" containerID="ffbd260bb4c6b631ff6750ac10fb6c86eac05a4ab7cb8dc2cda68297f1f6a7ba" Oct 30 19:12:04.597244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.581928 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:04.597244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.581984 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:04.652424 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.650426 4789 scope.go:122] "RemoveContainer" containerID="39a7be136d9e5c0d9d44fd01b245bd1e20819c2008fe85cfffc76327aff01a16" Oct 30 19:12:04.687189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.687061 4789 scope.go:122] "RemoveContainer" containerID="36053cc2722b6d0a72018386738240bec4c3af5cdfd0dab2de11b7bd3b50992f" Oct 30 19:12:04.817896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.817476 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:04.817896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.817530 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:04.863074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.862952 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:04.863074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:04.863004 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:05.076778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.076732 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 19:12:05.106227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.104315 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:05.410289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.409560 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerStarted","Data":"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01"} Oct 30 19:12:05.410289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.409640 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerStarted","Data":"80e0e6b4fb40eaf0c916ec6446af74b0ba77cb7c6635ff5de181c4dc9dd69eb6"} Oct 30 19:12:05.550230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.549312 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:05.588234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.587629 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:05.588234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.587713 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:05.821230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.820533 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:05.821230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.820598 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:05.864235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.862625 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:05.864235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:05.862683 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:06.561234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.558522 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:06.582232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.578580 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:06.582232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.578642 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:06.681173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.681113 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podStartSLOduration=4.6811003410000005 podStartE2EDuration="4.681100341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:05.583536833 +0000 UTC m=+1562.085447320" watchObservedRunningTime="2026-10-30 19:12:06.681100341 +0000 UTC m=+1563.183010826" Oct 30 19:12:06.821289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.820579 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:06.821289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.820640 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:06.862232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.858611 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:06.862232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:06.858674 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:07.459246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.459192 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:07.464798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.464494 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:12:07.578192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.578146 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:12:07.578741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.578196 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 19:12:07.827381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.827287 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:07.851337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.851289 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:07.862644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.862607 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:07.875478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:07.862906 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:08.668540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.668493 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:12:08.670171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.670143 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:12:08.671711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.671690 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:12:08.817476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.817438 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 19:12:08.817749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.817502 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 19:12:08.857275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.857242 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:12:08.857427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:08.857300 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:12:10.033000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.032966 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:10.034231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.034192 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:10.035058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.035033 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:12:10.220014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.219984 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:10.220494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.220443 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:10.221651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:10.221624 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:12:18.198061 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:12:18.197983 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:12:29.210289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.210247 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:12:29.214839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.214817 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247404 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247372 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247408 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247434 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247615 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpvd\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247663 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247754 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247807 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247913 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.247962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.247937 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.336255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.336204 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:12:29.349130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349099 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349179 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349241 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349268 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349354 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpvd\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349403 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349482 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349553 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349613 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349681 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.349801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.349706 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.350761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.350736 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.350912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.350738 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.351015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.350832 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.352361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.352327 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.352993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.352949 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.407253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.407229 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.407845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.407821 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpvd\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\") pod \"e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.529442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.529374 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:29.971788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.971760 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:12:29.975908 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:12:29.975881 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ce8fe9_fce6_4dba_93c7_4efd9712e9cf.slice/crio-e10ad03b30e11279e683f9e9d9dc3abd3f140eff12d13a315b201ba9c261f332 WatchSource:0}: Error finding container e10ad03b30e11279e683f9e9d9dc3abd3f140eff12d13a315b201ba9c261f332: Status 404 returned error can't find the container with id e10ad03b30e11279e683f9e9d9dc3abd3f140eff12d13a315b201ba9c261f332 Oct 30 19:12:29.977989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.977961 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:29.978054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.978023 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:29.978091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:29.978051 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:30.723680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:30.723642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" event={"ID":"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf","Type":"ContainerStarted","Data":"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c"} Oct 30 19:12:30.724106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:30.723708 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" event={"ID":"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf","Type":"ContainerStarted","Data":"e10ad03b30e11279e683f9e9d9dc3abd3f140eff12d13a315b201ba9c261f332"} Oct 30 19:12:30.932816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:30.932753 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podStartSLOduration=1.932738143 podStartE2EDuration="1.932738143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:30.8842962 +0000 UTC m=+1587.386206687" watchObservedRunningTime="2026-10-30 19:12:30.932738143 +0000 UTC m=+1587.434648629" Oct 30 19:12:31.530126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:31.530089 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:31.531836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:31.531809 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:31.531957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:31.531866 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:32.529994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:32.529957 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:32.530429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:32.530011 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:33.529718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:33.529678 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:33.529918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:33.529734 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:34.530653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:34.530618 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:34.531044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:34.530674 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:35.530099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:35.530066 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:35.530285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:35.530118 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:36.330874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.330834 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62"] Oct 30 19:12:36.336474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.336451 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.385876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.385845 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:36.390399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.390374 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62"] Oct 30 19:12:36.431742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.431712 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:12:36.431878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.431764 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:12:36.431878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.431870 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:12:36.457494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.457468 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbcq4\" (UniqueName: \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.457618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.457516 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.457727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.457708 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.529777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.529744 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:12:36.529908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.529787 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:12:36.558882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.558857 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbcq4\" (UniqueName: \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.558997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.558912 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.559066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.559044 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.559734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.559709 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.561524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.561502 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.619339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.619280 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbcq4\" (UniqueName: \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\") pod \"maas-api-revoke-sub-0c762ff1-lpd62\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:36.651934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:36.651904 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:37.063818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.058236 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62"] Oct 30 19:12:37.072486 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:12:37.072446 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6117c25_ecf2_4c99_a733_41e6fe21fd6b.slice/crio-33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c WatchSource:0}: Error finding container 33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c: Status 404 returned error can't find the container with id 33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c Oct 30 19:12:37.534148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.534125 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:37.534668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.534649 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:37.535196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.535176 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:12:37.781896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.781863 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" event={"ID":"a6117c25-ecf2-4c99-a733-41e6fe21fd6b","Type":"ContainerStarted","Data":"4f8bbb232dc38d5df4bb745fa99650b47cb46dd89f4d93df9132ccee5a9beb2d"} Oct 30 19:12:37.782060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.781924 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" event={"ID":"a6117c25-ecf2-4c99-a733-41e6fe21fd6b","Type":"ContainerStarted","Data":"33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c"} Oct 30 19:12:37.924985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:37.924935 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" podStartSLOduration=1.924920038 podStartE2EDuration="1.924920038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:37.92259642 +0000 UTC m=+1594.424506903" watchObservedRunningTime="2026-10-30 19:12:37.924920038 +0000 UTC m=+1594.426830525" Oct 30 19:12:38.790068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:38.790031 4789 generic.go:356] "Generic (PLEG): container finished" podID="a6117c25-ecf2-4c99-a733-41e6fe21fd6b" containerID="4f8bbb232dc38d5df4bb745fa99650b47cb46dd89f4d93df9132ccee5a9beb2d" exitCode=0 Oct 30 19:12:38.790486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:38.790111 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" event={"ID":"a6117c25-ecf2-4c99-a733-41e6fe21fd6b","Type":"ContainerDied","Data":"4f8bbb232dc38d5df4bb745fa99650b47cb46dd89f4d93df9132ccee5a9beb2d"} Oct 30 19:12:39.964044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:39.964020 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:40.004356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.004332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\" (UniqueName: \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\") pod \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " Oct 30 19:12:40.004506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.004471 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\") pod \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " Oct 30 19:12:40.004560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.004545 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\") pod \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\" (UID: \"a6117c25-ecf2-4c99-a733-41e6fe21fd6b\") " Oct 30 19:12:40.004889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.004859 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca" pod "a6117c25-ecf2-4c99-a733-41e6fe21fd6b" (UID: "a6117c25-ecf2-4c99-a733-41e6fe21fd6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:40.005129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.005103 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:40.006647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.006536 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca" pod "a6117c25-ecf2-4c99-a733-41e6fe21fd6b" (UID: "a6117c25-ecf2-4c99-a733-41e6fe21fd6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:40.006647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.006625 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4" pod "a6117c25-ecf2-4c99-a733-41e6fe21fd6b" (UID: "a6117c25-ecf2-4c99-a733-41e6fe21fd6b"). InnerVolumeSpecName "kube-api-access-mbcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:40.107250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.107154 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:40.107250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.107183 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbcq4\" (UniqueName: \"kubernetes.io/projected/a6117c25-ecf2-4c99-a733-41e6fe21fd6b-kube-api-access-mbcq4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:12:40.807390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.807358 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" Oct 30 19:12:40.807390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.807367 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62" event={"ID":"a6117c25-ecf2-4c99-a733-41e6fe21fd6b","Type":"ContainerDied","Data":"33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c"} Oct 30 19:12:40.807642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:40.807422 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33006e13f1630532ae1e4d4bfe2aa5f03e6ec7e3a88fdec1dd79ba0d9a57c52c" Oct 30 19:12:47.819234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:47.818327 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:47.910235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:47.906402 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:47.974261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:47.974223 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 19:12:48.110993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:48.110305 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 19:12:48.162495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:48.162458 4789 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:48.366626 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:12:48.366537 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:12:48.454320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:48.454280 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" podUID="29dbf484-abcb-4246-8e97-9784359be27c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:12:49.680096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.680059 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:12:49.681011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.680987 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a6117c25-ecf2-4c99-a733-41e6fe21fd6b" containerName="revoke-keys" Oct 30 19:12:49.681011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.681010 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6117c25-ecf2-4c99-a733-41e6fe21fd6b" containerName="revoke-keys" Oct 30 19:12:49.681195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.681184 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6117c25-ecf2-4c99-a733-41e6fe21fd6b" containerName="revoke-keys" Oct 30 19:12:49.685438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.685419 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.787959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.787926 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:12:49.801951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.801906 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c4ca9a-openshift-default-dockercfg-szvx2\"" Oct 30 19:12:49.859668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.859635 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.859860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.859763 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.859860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.859802 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.859860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.859825 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.859860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.859843 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.860087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.860062 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.860148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.860105 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.860202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.860170 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.860202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.860188 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6q56\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961554 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961657 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961699 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961780 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961811 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961900 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961928 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.961986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961970 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.962138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.961995 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6q56\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.962244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.962201 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.962346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.962271 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.962695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.962660 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.962862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.962845 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.963067 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.963050 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.964487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.964466 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:49.965123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:49.965101 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:50.039184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.036869 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:50.047911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.047762 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6q56\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\") pod \"e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:50.333339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.333259 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:50.795370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.795335 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:12:50.808112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.806921 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:50.808112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.807010 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:50.808112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.807055 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:12:50.936317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.936277 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerStarted","Data":"6381f94f425854393d57beb0ba5d363c96229d486447b124f23d2996d27b804f"} Oct 30 19:12:50.936496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:50.936388 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerStarted","Data":"3b4ffb8b43441f80412bd88bba434387ef5ebb6e44fca6c21ea840b6b4272cb3"} Oct 30 19:12:51.055403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:51.055303 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podStartSLOduration=2.055289738 podStartE2EDuration="2.055289738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:51.052042629 +0000 UTC m=+1607.553953118" watchObservedRunningTime="2026-10-30 19:12:51.055289738 +0000 UTC m=+1607.557200271" Oct 30 19:12:51.333924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:51.333835 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:51.339808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:51.339781 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:51.944589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:51.944558 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:51.945394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:51.945378 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:12:59.317238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.310116 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:12:59.362315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.359610 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.362315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.361503 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:12:59.436587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.436555 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c4ca9a\"" Oct 30 19:12:59.437014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.436966 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c4ca9a-dockercfg-27glf\"" Oct 30 19:12:59.488878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.488851 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc85b\" (UniqueName: \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.488986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.488957 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.590242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.590144 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc85b\" (UniqueName: \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.590242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.590198 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.590888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.590866 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.652693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.652668 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc85b\" (UniqueName: \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\") pod \"payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.695642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.695616 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:12:59.948474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.948447 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:12:59.953064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.953045 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994580 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994628 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994721 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994749 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994891 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.994987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994936 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5mt\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.995037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.994993 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.995129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.995115 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:12:59.995184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:12:59.995157 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.053435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.053402 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:13:00.084121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.084094 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:13:00.086189 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:00.086162 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e908c7_b0d4_49c4_bdff_d53a13521253.slice/crio-02077d7e7f88659d64073da8d4042eadea91de16f0c84708eb612257adfe8f87 WatchSource:0}: Error finding container 02077d7e7f88659d64073da8d4042eadea91de16f0c84708eb612257adfe8f87: Status 404 returned error can't find the container with id 02077d7e7f88659d64073da8d4042eadea91de16f0c84708eb612257adfe8f87 Oct 30 19:13:00.096126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096092 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.096250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096201 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.096320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096271 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5mt\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.096320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096299 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.096425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096370 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.096425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.096410 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.097520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.097500 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.097640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.097535 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.097808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.097781 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.099261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.098238 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.099261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.098281 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.099261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.098316 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.099261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.098660 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.099261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.099199 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.106010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.105987 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.107094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.107057 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.157136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.157113 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5mt\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.157354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.157336 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.269628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.269553 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:00.722730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.722699 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:13:00.733258 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:00.726995 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fbdd87_af90_4c1f_96b5_702155371343.slice/crio-704e5649e83737897a19e0affbf856fcb63f385f1f1c37c3d7cf0686f7ec226a WatchSource:0}: Error finding container 704e5649e83737897a19e0affbf856fcb63f385f1f1c37c3d7cf0686f7ec226a: Status 404 returned error can't find the container with id 704e5649e83737897a19e0affbf856fcb63f385f1f1c37c3d7cf0686f7ec226a Oct 30 19:13:00.733258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.730412 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:00.733258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.730498 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:00.733258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:00.730541 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:01.021054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.020278 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" event={"ID":"64e908c7-b0d4-49c4-bdff-d53a13521253","Type":"ContainerStarted","Data":"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f"} Oct 30 19:13:01.021054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.020351 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" event={"ID":"64e908c7-b0d4-49c4-bdff-d53a13521253","Type":"ContainerStarted","Data":"02077d7e7f88659d64073da8d4042eadea91de16f0c84708eb612257adfe8f87"} Oct 30 19:13:01.021054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.020397 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:13:01.022358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.022327 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" event={"ID":"69fbdd87-af90-4c1f-96b5-702155371343","Type":"ContainerStarted","Data":"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed"} Oct 30 19:13:01.022498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.022372 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" event={"ID":"69fbdd87-af90-4c1f-96b5-702155371343","Type":"ContainerStarted","Data":"704e5649e83737897a19e0affbf856fcb63f385f1f1c37c3d7cf0686f7ec226a"} Oct 30 19:13:01.138824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.138767 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" podStartSLOduration=2.138751701 podStartE2EDuration="2.138751701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:01.134747714 +0000 UTC m=+1617.636658200" watchObservedRunningTime="2026-10-30 19:13:01.138751701 +0000 UTC m=+1617.640662187" Oct 30 19:13:01.258809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.258759 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podStartSLOduration=2.258744733 podStartE2EDuration="2.258744733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:01.255643137 +0000 UTC m=+1617.757553620" watchObservedRunningTime="2026-10-30 19:13:01.258744733 +0000 UTC m=+1617.760655284" Oct 30 19:13:01.269673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.269644 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:01.270873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.270843 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:01.270963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:01.270884 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:02.270829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:02.270794 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:02.271355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:02.270850 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:03.270559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:03.270532 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:03.270787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:03.270765 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:04.269893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:04.269857 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:04.270423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:04.270386 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:05.270579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:05.270543 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:05.270984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:05.270594 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:06.061992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:06.061961 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x"] Oct 30 19:13:06.160267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:06.158798 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0711725c-9v75x"] Oct 30 19:13:06.270631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:06.270596 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:06.271022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:06.270646 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:07.270242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.270184 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:13:07.270408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.270269 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:13:07.448678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.448639 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:13:07.454005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.453978 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.543294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.543199 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:13:07.560827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.560766 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-942c5e85\"/\"openshift-service-ca.crt\"" Oct 30 19:13:07.560827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.560813 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-942c5e85\"/\"kube-root-ca.crt\"" Oct 30 19:13:07.574090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574061 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.574228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574102 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.574228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574131 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4g7\" (UniqueName: \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.574319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574296 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.574431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574412 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.574522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.574508 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.588131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.588106 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-942c5e85\"/\"default-dockercfg-x2qtr\"" Oct 30 19:13:07.588131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.588106 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-942c5e85\"/\"e2e-lifecycle-model-942c5e85-kserve-self-signed-certs\"" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679058 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679287 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679406 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679435 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679465 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4g7\" (UniqueName: \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.680104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.679531 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.681245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.681022 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.682043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.681707 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.682523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.682501 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.684671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.684605 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.692599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.692349 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b556f1e-cdde-41e2-8ff5-6c5e6abacb04" path="/var/lib/kubelet/pods/6b556f1e-cdde-41e2-8ff5-6c5e6abacb04/volumes" Oct 30 19:13:07.700109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.700060 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.762944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.762910 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4g7\" (UniqueName: \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\") pod \"e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:07.767638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:07.767395 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:08.578984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:08.578955 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:13:08.742335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:08.742246 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:08.743155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:08.743131 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:08.743967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:08.743946 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:13:09.268755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:09.268712 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" event={"ID":"7e4a6351-6a80-46da-a49e-5d72566ee749","Type":"ContainerStarted","Data":"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280"} Oct 30 19:13:09.268932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:09.268760 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" event={"ID":"7e4a6351-6a80-46da-a49e-5d72566ee749","Type":"ContainerStarted","Data":"73551bb35ae53a3e20fd2f3b075ac0059438e8cb9a460bee64a0d8c25ed7a021"} Oct 30 19:13:09.390504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:09.390448 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" podStartSLOduration=2.3904295380000002 podStartE2EDuration="2.390429538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:09.384561909 +0000 UTC m=+1625.886472386" watchObservedRunningTime="2026-10-30 19:13:09.390429538 +0000 UTC m=+1625.892340027" Oct 30 19:13:09.768511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:09.767967 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:12.047119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.047091 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:13:12.296955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.296925 4789 generic.go:356] "Generic (PLEG): container finished" podID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerID="7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01" exitCode=0 Oct 30 19:13:12.297166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.297009 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerDied","Data":"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01"} Oct 30 19:13:12.297444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.297427 4789 scope.go:122] "RemoveContainer" containerID="7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01" Oct 30 19:13:12.298980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.298951 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:12.299044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.299024 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:12.299089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:12.299054 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:13.324490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:13.324457 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerStarted","Data":"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2"} Oct 30 19:13:13.538679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:13.538641 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:13:13.538915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:13.538889 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:13:13.538971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:13.538951 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:13:14.538393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:14.538361 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:13:14.538774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:14.538411 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:13:15.538543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:15.538505 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:13:15.539026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:15.538573 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:13:16.538939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:16.538908 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:13:16.539366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:16.538971 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:13:17.539056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.539023 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:13:17.539522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.539084 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 19:13:17.560960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.560933 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr"] Oct 30 19:13:17.564340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.564317 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.627408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.627377 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr"] Oct 30 19:13:17.645898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.645863 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:13:17.665111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.665081 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:17.665251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.665161 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:13:17.665251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.665183 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:13:17.672121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.672097 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nps6d\" (UniqueName: \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.672233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.672136 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.672277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.672251 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.768753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.768721 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:17.773812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.773784 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nps6d\" (UniqueName: \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.774036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.774016 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.774158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.774137 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.774745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.774720 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.776485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.776462 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.781814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.781796 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:17.832827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.832758 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nps6d\" (UniqueName: \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\") pod \"maas-api-revoke-sub-7e79953b-bnmqr\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:17.877487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:17.877468 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:18.198535 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:13:18.198504 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:13:18.309670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.309628 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr"] Oct 30 19:13:18.316171 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:18.316141 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a5a7e2_5908_475c_bb49_c6862157cbff.slice/crio-2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720 WatchSource:0}: Error finding container 2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720: Status 404 returned error can't find the container with id 2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720 Oct 30 19:13:18.366454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.366416 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" event={"ID":"f6a5a7e2-5908-475c-bb49-c6862157cbff","Type":"ContainerStarted","Data":"2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720"} Oct 30 19:13:18.378063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.378035 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:13:18.543723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.543694 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:13:18.544203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.544184 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:13:18.544897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:18.544877 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:13:19.070026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:19.069995 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4"] Oct 30 19:13:19.074370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:19.074347 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3ffb00-l67f4"] Oct 30 19:13:19.378221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:19.378054 4789 generic.go:356] "Generic (PLEG): container finished" podID="f6a5a7e2-5908-475c-bb49-c6862157cbff" containerID="05209479dc6cd1d9b282f3cc23c0287795391f469fce9d11a03d5c7758197a43" exitCode=0 Oct 30 19:13:19.378221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:19.378097 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" event={"ID":"f6a5a7e2-5908-475c-bb49-c6862157cbff","Type":"ContainerDied","Data":"05209479dc6cd1d9b282f3cc23c0287795391f469fce9d11a03d5c7758197a43"} Oct 30 19:13:19.669006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:19.668932 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1586106-aa3e-4956-81b0-42923f9014b5" path="/var/lib/kubelet/pods/b1586106-aa3e-4956-81b0-42923f9014b5/volumes" Oct 30 19:13:20.546117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.546092 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:20.614315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.614278 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\") pod \"f6a5a7e2-5908-475c-bb49-c6862157cbff\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " Oct 30 19:13:20.614437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.614409 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\" (UniqueName: \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\") pod \"f6a5a7e2-5908-475c-bb49-c6862157cbff\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " Oct 30 19:13:20.614523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.614509 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\") pod \"f6a5a7e2-5908-475c-bb49-c6862157cbff\" (UID: \"f6a5a7e2-5908-475c-bb49-c6862157cbff\") " Oct 30 19:13:20.614852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.614828 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca" pod "f6a5a7e2-5908-475c-bb49-c6862157cbff" (UID: "f6a5a7e2-5908-475c-bb49-c6862157cbff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:20.614969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.614954 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:20.616309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.616282 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca" pod "f6a5a7e2-5908-475c-bb49-c6862157cbff" (UID: "f6a5a7e2-5908-475c-bb49-c6862157cbff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:20.616475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.616457 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d" pod "f6a5a7e2-5908-475c-bb49-c6862157cbff" (UID: "f6a5a7e2-5908-475c-bb49-c6862157cbff"). InnerVolumeSpecName "kube-api-access-nps6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:20.716080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.716050 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nps6d\" (UniqueName: \"kubernetes.io/projected/f6a5a7e2-5908-475c-bb49-c6862157cbff-kube-api-access-nps6d\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:20.716080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:20.716082 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a5a7e2-5908-475c-bb49-c6862157cbff-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:21.395650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:21.395612 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" event={"ID":"f6a5a7e2-5908-475c-bb49-c6862157cbff","Type":"ContainerDied","Data":"2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720"} Oct 30 19:13:21.395650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:21.395641 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr" Oct 30 19:13:21.395847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:21.395670 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3918801b510b3550d3ef4242fed503f3628985a553ccc83870150ec38b3720" Oct 30 19:13:42.097334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.097300 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6"] Oct 30 19:13:42.098138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.098117 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f6a5a7e2-5908-475c-bb49-c6862157cbff" containerName="revoke-keys" Oct 30 19:13:42.098138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.098138 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a5a7e2-5908-475c-bb49-c6862157cbff" containerName="revoke-keys" Oct 30 19:13:42.098350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.098340 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a5a7e2-5908-475c-bb49-c6862157cbff" containerName="revoke-keys" Oct 30 19:13:42.103120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.103100 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.156761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.156729 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:42.160556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.160532 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6"] Oct 30 19:13:42.178434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.178407 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.178536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.178490 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.178601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.178583 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd58n\" (UniqueName: \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.201485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.201462 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:13:42.201485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.201462 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:13:42.201661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.201590 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:13:42.279272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.279242 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.279411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.279312 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.279411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.279355 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd58n\" (UniqueName: \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.279962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.279937 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.281667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.281644 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.337867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.337812 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd58n\" (UniqueName: \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\") pod \"maas-api-revoke-sub-6d3acba7-r9qc6\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.418174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.418116 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:42.805545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:42.805519 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6"] Oct 30 19:13:42.810912 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:42.810882 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951b817b_f618_488b_9b16_a80c22a0fec2.slice/crio-dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678 WatchSource:0}: Error finding container dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678: Status 404 returned error can't find the container with id dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678 Oct 30 19:13:43.576304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:43.576269 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" event={"ID":"951b817b-f618-488b-9b16-a80c22a0fec2","Type":"ContainerStarted","Data":"d5d00260037bb7f398d805cf8849841d4537746c9674d69b322799332164e9ae"} Oct 30 19:13:43.576657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:43.576334 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" event={"ID":"951b817b-f618-488b-9b16-a80c22a0fec2","Type":"ContainerStarted","Data":"dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678"} Oct 30 19:13:43.691843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:43.691777 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" podStartSLOduration=1.691760708 podStartE2EDuration="1.691760708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:43.685772395 +0000 UTC m=+1660.187682881" watchObservedRunningTime="2026-10-30 19:13:43.691760708 +0000 UTC m=+1660.193671196" Oct 30 19:13:44.587483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:44.587437 4789 generic.go:356] "Generic (PLEG): container finished" podID="951b817b-f618-488b-9b16-a80c22a0fec2" containerID="d5d00260037bb7f398d805cf8849841d4537746c9674d69b322799332164e9ae" exitCode=0 Oct 30 19:13:44.587850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:44.587520 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" event={"ID":"951b817b-f618-488b-9b16-a80c22a0fec2","Type":"ContainerDied","Data":"d5d00260037bb7f398d805cf8849841d4537746c9674d69b322799332164e9ae"} Oct 30 19:13:45.811866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.811841 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:45.927009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.926921 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\") pod \"951b817b-f618-488b-9b16-a80c22a0fec2\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " Oct 30 19:13:45.927153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.927008 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\" (UniqueName: \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\") pod \"951b817b-f618-488b-9b16-a80c22a0fec2\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " Oct 30 19:13:45.927153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.927080 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\") pod \"951b817b-f618-488b-9b16-a80c22a0fec2\" (UID: \"951b817b-f618-488b-9b16-a80c22a0fec2\") " Oct 30 19:13:45.927919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.927896 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca" pod "951b817b-f618-488b-9b16-a80c22a0fec2" (UID: "951b817b-f618-488b-9b16-a80c22a0fec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:45.933672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.930543 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n" pod "951b817b-f618-488b-9b16-a80c22a0fec2" (UID: "951b817b-f618-488b-9b16-a80c22a0fec2"). InnerVolumeSpecName "kube-api-access-kd58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:45.942278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:45.934522 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca" pod "951b817b-f618-488b-9b16-a80c22a0fec2" (UID: "951b817b-f618-488b-9b16-a80c22a0fec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:46.028920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.028882 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:46.028920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.028922 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd58n\" (UniqueName: \"kubernetes.io/projected/951b817b-f618-488b-9b16-a80c22a0fec2-kube-api-access-kd58n\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:46.029146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.028940 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951b817b-f618-488b-9b16-a80c22a0fec2-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:46.253648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.253612 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:13:46.254456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.254431 4789 cpu_manager.go:414] "containerMap: removing container" podUID="951b817b-f618-488b-9b16-a80c22a0fec2" containerName="revoke-keys" Oct 30 19:13:46.254456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.254454 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951b817b-f618-488b-9b16-a80c22a0fec2" containerName="revoke-keys" Oct 30 19:13:46.254657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.254643 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="951b817b-f618-488b-9b16-a80c22a0fec2" containerName="revoke-keys" Oct 30 19:13:46.258928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.258896 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.321731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.321695 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f1b38f-openshift-default-dockercfg-6xmnz\"" Oct 30 19:13:46.585326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.585238 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.585575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.585553 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.585955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.585933 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586083 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586247 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586366 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586656 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586781 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4sk\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.586903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.586889 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.589477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.589456 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:13:46.668439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.668387 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" event={"ID":"951b817b-f618-488b-9b16-a80c22a0fec2","Type":"ContainerDied","Data":"dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678"} Oct 30 19:13:46.668725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.668707 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd9aa7964c1e556ceabee8f46dbbe1b8d2e8f41fb7bb919e4f70649ce8e20678" Oct 30 19:13:46.668909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.668897 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6" Oct 30 19:13:46.704554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.689298 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.704855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.704829 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705001 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705105 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705277 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705411 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4sk\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705524 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705680 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.705809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.705797 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.747245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.712298 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.747245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.712611 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.747245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.712879 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.747245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.713574 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.747245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.728261 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.799237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.788171 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.799237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.799086 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.923165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.920170 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:46.923165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:46.920848 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4sk\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\") pod \"e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:47.211520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:47.211462 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:48.068179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.068149 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:13:48.105405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.105352 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:48.105655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.105632 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:48.105823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.105803 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:48.355248 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:13:48.352450 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:13:48.945826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.945790 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerStarted","Data":"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8"} Oct 30 19:13:48.946018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:48.945851 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerStarted","Data":"fdb4f94329bacd3fcea3d1b4ee1beb853914cb3a397a2331684b7781280ba4f1"} Oct 30 19:13:49.115636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:49.115583 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podStartSLOduration=3.115568094 podStartE2EDuration="3.115568094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:49.06770906 +0000 UTC m=+1665.569619546" watchObservedRunningTime="2026-10-30 19:13:49.115568094 +0000 UTC m=+1665.617478580" Oct 30 19:13:49.217913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:49.212290 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:49.220987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:49.220945 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:49.955985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:49.955952 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:49.957040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:49.957017 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:13:53.573389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.573358 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:13:53.577794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.577775 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.666355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.649943 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f44636-openshift-default-dockercfg-s2jb5\"" Oct 30 19:13:53.748455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.748406 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.776510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.772777 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.776510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.774504 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.776510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.774648 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.780551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.780516 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.793505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.793471 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.806227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.793970 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.806227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.803551 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.806227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.803728 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grhhv\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.818408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.818370 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:13:53.906464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906436 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906607 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906671 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906764 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grhhv\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906830 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.906841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.906831 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.908992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.908961 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.909284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909259 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.911379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909383 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.911379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909510 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.911379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909557 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.911379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909677 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.911379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.909923 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.921562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.921535 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.928746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.928716 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.975283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.975236 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grhhv\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:53.980978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:53.980945 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\") pod \"e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:54.191829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:54.191796 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:54.659904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:54.659824 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:13:54.692902 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:54.692875 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5836ee2e_5b1b_4c2c_a751_c5efaa41cc05.slice/crio-9849c260f487f2c80baddcafbe077013edae6b4142f9c5db3d2fadd8bdc471c8 WatchSource:0}: Error finding container 9849c260f487f2c80baddcafbe077013edae6b4142f9c5db3d2fadd8bdc471c8: Status 404 returned error can't find the container with id 9849c260f487f2c80baddcafbe077013edae6b4142f9c5db3d2fadd8bdc471c8 Oct 30 19:13:54.695449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:54.695407 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:54.695555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:54.695512 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:54.695612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:54.695562 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:13:55.000796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.000754 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerStarted","Data":"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937"} Oct 30 19:13:55.000969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.000825 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerStarted","Data":"9849c260f487f2c80baddcafbe077013edae6b4142f9c5db3d2fadd8bdc471c8"} Oct 30 19:13:55.119338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.119281 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podStartSLOduration=2.119267405 podStartE2EDuration="2.119267405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:55.115836622 +0000 UTC m=+1671.617747108" watchObservedRunningTime="2026-10-30 19:13:55.119267405 +0000 UTC m=+1671.621177890" Oct 30 19:13:55.191964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.191936 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:55.197191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.197164 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:55.935703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.935676 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:13:55.940610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.940589 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:55.995778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.995738 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f1b38f\"" Oct 30 19:13:55.995778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.995738 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f1b38f-dockercfg-tpdd2\"" Oct 30 19:13:55.998920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:55.998891 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:13:56.009530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.009507 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:56.010437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.010420 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:13:56.053148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.053120 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.053348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.053325 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqv9g\" (UniqueName: \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.158231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.157782 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.158231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.157912 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqv9g\" (UniqueName: \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.162229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.160002 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.219988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.219716 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqv9g\" (UniqueName: \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\") pod \"payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.255828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.255796 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:56.643058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:56.642977 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:13:56.648647 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:13:56.648611 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e8697f_ee19_4a88_825f_5dc40fe8d587.slice/crio-2458555b28f2b9840c1c508de53be0b5d9028441151a007bb59753c49f115db9 WatchSource:0}: Error finding container 2458555b28f2b9840c1c508de53be0b5d9028441151a007bb59753c49f115db9: Status 404 returned error can't find the container with id 2458555b28f2b9840c1c508de53be0b5d9028441151a007bb59753c49f115db9 Oct 30 19:13:57.017965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:57.017929 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" event={"ID":"e4e8697f-ee19-4a88-825f-5dc40fe8d587","Type":"ContainerStarted","Data":"e056f10d63a2067ead92a527b44cbcb1d2b0f234c1bf8fc5a936ff948fcb0dab"} Oct 30 19:13:57.018386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:57.017981 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" event={"ID":"e4e8697f-ee19-4a88-825f-5dc40fe8d587","Type":"ContainerStarted","Data":"2458555b28f2b9840c1c508de53be0b5d9028441151a007bb59753c49f115db9"} Oct 30 19:13:57.018386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:57.018022 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:13:57.306328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:13:57.306230 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" podStartSLOduration=2.306200447 podStartE2EDuration="2.306200447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:57.304537917 +0000 UTC m=+1673.806448425" watchObservedRunningTime="2026-10-30 19:13:57.306200447 +0000 UTC m=+1673.808110932" Oct 30 19:14:05.052759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:05.052711 4789 scope.go:122] "RemoveContainer" containerID="aa40a65f163b692482d4517b276f87fca3ebe179f02afcaf9314b448b30ad51e" Oct 30 19:14:08.028769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:08.028739 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:14:18.198896 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:14:18.198856 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:14:19.413615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.413524 4789 generic.go:356] "Generic (PLEG): container finished" podID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerID="6381f94f425854393d57beb0ba5d363c96229d486447b124f23d2996d27b804f" exitCode=0 Oct 30 19:14:19.413615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.413602 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerDied","Data":"6381f94f425854393d57beb0ba5d363c96229d486447b124f23d2996d27b804f"} Oct 30 19:14:19.414106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.414090 4789 scope.go:122] "RemoveContainer" containerID="6381f94f425854393d57beb0ba5d363c96229d486447b124f23d2996d27b804f" Oct 30 19:14:19.415721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.415695 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:14:19.415799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.415763 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:14:19.415836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:19.415797 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:14:20.431727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:20.431695 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerStarted","Data":"b34857e992f063d8c6073b6ef3f10f96d37ea8db9f8861b9b0533152e8e5a1f2"} Oct 30 19:14:21.333694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:21.333655 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:14:21.335159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:21.335131 4789 patch_prober.go:29] interesting pod/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:14:21.335315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:21.335194 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:14:21.944721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:21.944687 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:14:22.334322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:22.334202 4789 patch_prober.go:29] interesting pod/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:14:22.334322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:22.334302 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:14:23.335668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:23.335628 4789 patch_prober.go:29] interesting pod/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:14:23.336198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:23.335690 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:14:24.334171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:24.334131 4789 patch_prober.go:29] interesting pod/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:14:24.334367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:24.334204 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:14:25.334156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:25.334109 4789 patch_prober.go:29] interesting pod/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:14:25.334581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:25.334182 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:14:26.339172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:26.339138 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:14:26.340336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:26.340316 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:14:29.060926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:29.060887 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7"] Oct 30 19:14:29.107374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:29.107337 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b03ad442-szrj7"] Oct 30 19:14:29.675364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:29.675329 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2" path="/var/lib/kubelet/pods/0b601f1b-007d-4f2b-aadd-9a48fb3d1ea2/volumes" Oct 30 19:14:35.135957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.135923 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:14:35.141582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.141561 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.204472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.204436 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:14:35.206039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.206013 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2g8\" (UniqueName: \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.206182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.206162 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.236921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.236190 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0d2ef6-dockercfg-v425b\"" Oct 30 19:14:35.236921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.236880 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0d2ef6\"" Oct 30 19:14:35.307329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.307294 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.307537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.307509 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2g8\" (UniqueName: \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.308802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.308777 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.377227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.375155 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2g8\" (UniqueName: \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\") pod \"payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.455422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.455391 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:35.843332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:35.843250 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:14:35.846916 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:14:35.846886 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b54b94e_704c_470b_8f09_0973f183f4c9.slice/crio-78058d3f2af3668a5921349cc5f14d34d4e20466bcc09f7880c51f20ab549267 WatchSource:0}: Error finding container 78058d3f2af3668a5921349cc5f14d34d4e20466bcc09f7880c51f20ab549267: Status 404 returned error can't find the container with id 78058d3f2af3668a5921349cc5f14d34d4e20466bcc09f7880c51f20ab549267 Oct 30 19:14:36.565263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:36.565226 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" event={"ID":"8b54b94e-704c-470b-8f09-0973f183f4c9","Type":"ContainerStarted","Data":"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80"} Oct 30 19:14:36.565740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:36.565299 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:36.565740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:36.565325 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" event={"ID":"8b54b94e-704c-470b-8f09-0973f183f4c9","Type":"ContainerStarted","Data":"78058d3f2af3668a5921349cc5f14d34d4e20466bcc09f7880c51f20ab549267"} Oct 30 19:14:36.677274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:36.677201 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" podStartSLOduration=1.677187028 podStartE2EDuration="1.677187028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:36.675498543 +0000 UTC m=+1713.177409028" watchObservedRunningTime="2026-10-30 19:14:36.677187028 +0000 UTC m=+1713.179097573" Oct 30 19:14:47.576831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:14:47.576799 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:14:48.197087 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:14:48.197047 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:15:00.171280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.171244 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:00.175669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.175645 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:00.175806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.175783 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.180357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.180333 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:00.180535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.180431 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.184532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.184513 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.217834 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.217804 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:00.223064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.223043 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.223277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.223232 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:00.227931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.227903 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.228194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.228179 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:00.230248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.230196 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:00.232317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.232294 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.232711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.232691 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:00.232799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.232761 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:00.237397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.237379 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.272467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.262564 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:15:00.292667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.281125 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.318235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.311359 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.318235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.311473 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.318235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.311552 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb5s\" (UniqueName: \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.318235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.311663 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdt7\" (UniqueName: \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.318235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.311708 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb9c\" (UniqueName: \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.414079 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb5s\" (UniqueName: \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.414921 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdt7\" (UniqueName: \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.414979 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nnb\" (UniqueName: \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415017 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb9c\" (UniqueName: \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415084 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415111 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415157 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6wk\" (UniqueName: \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415181 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415274 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415327 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxdr\" (UniqueName: \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415385 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415456 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415522 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.415557 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ms5\" (UniqueName: \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.417243 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.417956 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.431286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.419870 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.452149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.442572 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:00.452736 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.452655 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:00.462132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.461986 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:00.478309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.478282 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:00.484267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.484073 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdt7\" (UniqueName: \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.486796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.486766 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb5s\" (UniqueName: \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.487844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.487814 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb9c\" (UniqueName: \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.488276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.488252 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:00.494610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.494409 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:00.510470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.510416 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:00.516970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.515472 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:00.521746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.521673 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.521870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.521764 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ms5\" (UniqueName: \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.521927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.521888 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nnb\" (UniqueName: \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.522066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.521989 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.522066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.522041 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6wk\" (UniqueName: \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.522184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.522072 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.522184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.522159 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.522294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.522195 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxdr\" (UniqueName: \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.527710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.524988 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.527710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.525651 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.527710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.527089 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.530732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.528226 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:00.530732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.529013 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.627230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.621360 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6wk\" (UniqueName: \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\") pod \"maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.627230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.624155 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxdr\" (UniqueName: \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\") pod \"maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:00.638232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.634984 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nnb\" (UniqueName: \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\") pod \"maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.638232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.635615 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ms5\" (UniqueName: \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.638896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.638869 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:00.671481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.671448 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:00.840195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.840164 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:00.850381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.850174 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:00.888393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:00.885658 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:01.235178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.234653 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:01.336193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.336113 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:01.339009 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:01.338979 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7db509_565e_4310_808c_06420c73cd0b.slice/crio-8cbafaab1f548b3dcd1f0f5b88d2e2ffc9f0b9ba876bc3004db30206de05a006 WatchSource:0}: Error finding container 8cbafaab1f548b3dcd1f0f5b88d2e2ffc9f0b9ba876bc3004db30206de05a006: Status 404 returned error can't find the container with id 8cbafaab1f548b3dcd1f0f5b88d2e2ffc9f0b9ba876bc3004db30206de05a006 Oct 30 19:15:01.384494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.384376 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:01.389711 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:01.389676 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23b3e98_f0cc_455d_a226_30984fa249d6.slice/crio-ca2165a1514dd647ed6bb99ddb67a6bd0440587a54872d24294b5d7585a59a6f WatchSource:0}: Error finding container ca2165a1514dd647ed6bb99ddb67a6bd0440587a54872d24294b5d7585a59a6f: Status 404 returned error can't find the container with id ca2165a1514dd647ed6bb99ddb67a6bd0440587a54872d24294b5d7585a59a6f Oct 30 19:15:01.391117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.391091 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:01.393271 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:01.393205 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd89999b_7717_4ed8_a91e_4eb905c012bd.slice/crio-2a1b754ddb0f6e9d0efa63771ca34576e62d6282aedd2cc220a5776a2f8d446c WatchSource:0}: Error finding container 2a1b754ddb0f6e9d0efa63771ca34576e62d6282aedd2cc220a5776a2f8d446c: Status 404 returned error can't find the container with id 2a1b754ddb0f6e9d0efa63771ca34576e62d6282aedd2cc220a5776a2f8d446c Oct 30 19:15:01.460943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.460907 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:01.470640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.470594 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:01.533731 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:01.533654 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4258318d_b0c1_4dee_a808_719d06c12aa7.slice/crio-a1bcc9e5a69bab8e853a775cf56bf54fc71d76c7fb2c923eac9e6598b6693eaa WatchSource:0}: Error finding container a1bcc9e5a69bab8e853a775cf56bf54fc71d76c7fb2c923eac9e6598b6693eaa: Status 404 returned error can't find the container with id a1bcc9e5a69bab8e853a775cf56bf54fc71d76c7fb2c923eac9e6598b6693eaa Oct 30 19:15:01.534014 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:01.533984 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb696126_0e4b_489d_903c_191c12bcf922.slice/crio-ffd5167c3f6cec35ec16d5eacda04adf643a10a8fbe446851497e5e59b19dde8 WatchSource:0}: Error finding container ffd5167c3f6cec35ec16d5eacda04adf643a10a8fbe446851497e5e59b19dde8: Status 404 returned error can't find the container with id ffd5167c3f6cec35ec16d5eacda04adf643a10a8fbe446851497e5e59b19dde8 Oct 30 19:15:01.537434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.537042 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:01.775612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.775532 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerStarted","Data":"a1bcc9e5a69bab8e853a775cf56bf54fc71d76c7fb2c923eac9e6598b6693eaa"} Oct 30 19:15:01.776818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.776794 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerStarted","Data":"ca2165a1514dd647ed6bb99ddb67a6bd0440587a54872d24294b5d7585a59a6f"} Oct 30 19:15:01.777907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.777859 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" event={"ID":"fb696126-0e4b-489d-903c-191c12bcf922","Type":"ContainerStarted","Data":"ffd5167c3f6cec35ec16d5eacda04adf643a10a8fbe446851497e5e59b19dde8"} Oct 30 19:15:01.779001 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.778980 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerStarted","Data":"8f11a62a6c419d25a069d25369940f4a1a354ad77d48b7a5e8941cbe39b77a1d"} Oct 30 19:15:01.780390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.780345 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerStarted","Data":"2a1b754ddb0f6e9d0efa63771ca34576e62d6282aedd2cc220a5776a2f8d446c"} Oct 30 19:15:01.781634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.781612 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerStarted","Data":"8cbafaab1f548b3dcd1f0f5b88d2e2ffc9f0b9ba876bc3004db30206de05a006"} Oct 30 19:15:01.783282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:01.783260 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" event={"ID":"b5bc77e1-ff36-4f87-9f36-b8174fce0171","Type":"ContainerStarted","Data":"a6e83a29318d0b65b2a2559eb5f071071376080b3ece5549104371fb6d5960a4"} Oct 30 19:15:02.823463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.820033 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerStarted","Data":"c6d253d48ae40249456e0241116e60d7a895648a5e8a35ffb38387ada55be8dc"} Oct 30 19:15:02.852354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.827884 4789 generic.go:356] "Generic (PLEG): container finished" podID="fb696126-0e4b-489d-903c-191c12bcf922" containerID="88ae3ebeaa32546897f41cd6e59e5ce727ddddb6b6723139804bd05ab5b360b3" exitCode=60 Oct 30 19:15:02.852354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.828018 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" event={"ID":"fb696126-0e4b-489d-903c-191c12bcf922","Type":"ContainerDied","Data":"88ae3ebeaa32546897f41cd6e59e5ce727ddddb6b6723139804bd05ab5b360b3"} Oct 30 19:15:02.855744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.855715 4789 scope.go:122] "RemoveContainer" containerID="88ae3ebeaa32546897f41cd6e59e5ce727ddddb6b6723139804bd05ab5b360b3" Oct 30 19:15:02.861283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.860920 4789 generic.go:356] "Generic (PLEG): container finished" podID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerID="1e3f4a1d2486c6a6b2a07c5ae62c74645e148c44dccb6f6f00968c6f07b7652a" exitCode=60 Oct 30 19:15:02.861283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.861084 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerDied","Data":"1e3f4a1d2486c6a6b2a07c5ae62c74645e148c44dccb6f6f00968c6f07b7652a"} Oct 30 19:15:02.861671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.861648 4789 scope.go:122] "RemoveContainer" containerID="1e3f4a1d2486c6a6b2a07c5ae62c74645e148c44dccb6f6f00968c6f07b7652a" Oct 30 19:15:02.866737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.866713 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerID="ddbad1b6a3eba27ac58f04f288fdfa2e41b7b7e9ed7b4ceff1dec7962b15f3f4" exitCode=60 Oct 30 19:15:02.866834 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.866782 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerDied","Data":"ddbad1b6a3eba27ac58f04f288fdfa2e41b7b7e9ed7b4ceff1dec7962b15f3f4"} Oct 30 19:15:02.867036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.867019 4789 scope.go:122] "RemoveContainer" containerID="ddbad1b6a3eba27ac58f04f288fdfa2e41b7b7e9ed7b4ceff1dec7962b15f3f4" Oct 30 19:15:02.870143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.870115 4789 generic.go:356] "Generic (PLEG): container finished" podID="3c7db509-565e-4310-808c-06420c73cd0b" containerID="5be44879a922cb8b4ab0e7def447556cdd70293f318bd5927378f6891cfa7740" exitCode=60 Oct 30 19:15:02.870278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.870203 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerDied","Data":"5be44879a922cb8b4ab0e7def447556cdd70293f318bd5927378f6891cfa7740"} Oct 30 19:15:02.870582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.870534 4789 scope.go:122] "RemoveContainer" containerID="5be44879a922cb8b4ab0e7def447556cdd70293f318bd5927378f6891cfa7740" Oct 30 19:15:02.873620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.873597 4789 generic.go:356] "Generic (PLEG): container finished" podID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerID="eeba5a9d992a53ea7929f3a6c66da508f5e79f0c88bdd02efb914dc2b7ea617b" exitCode=60 Oct 30 19:15:02.874069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.874043 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" event={"ID":"b5bc77e1-ff36-4f87-9f36-b8174fce0171","Type":"ContainerDied","Data":"eeba5a9d992a53ea7929f3a6c66da508f5e79f0c88bdd02efb914dc2b7ea617b"} Oct 30 19:15:02.874710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.874630 4789 scope.go:122] "RemoveContainer" containerID="eeba5a9d992a53ea7929f3a6c66da508f5e79f0c88bdd02efb914dc2b7ea617b" Oct 30 19:15:02.877167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.877149 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerStarted","Data":"6ba96477570f3534279424647fc990c8a03707ca013c4f7ceffa152a681fae3c"} Oct 30 19:15:02.877412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:02.877399 4789 scope.go:122] "RemoveContainer" containerID="6ba96477570f3534279424647fc990c8a03707ca013c4f7ceffa152a681fae3c" Oct 30 19:15:03.053243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.051565 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" podStartSLOduration=3.051544065 podStartE2EDuration="3.051544065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.933591329 +0000 UTC m=+1739.435501828" watchObservedRunningTime="2026-10-30 19:15:03.051544065 +0000 UTC m=+1739.553454551" Oct 30 19:15:03.981292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.981261 4789 generic.go:356] "Generic (PLEG): container finished" podID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerID="6ba96477570f3534279424647fc990c8a03707ca013c4f7ceffa152a681fae3c" exitCode=60 Oct 30 19:15:03.981656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.981425 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerDied","Data":"6ba96477570f3534279424647fc990c8a03707ca013c4f7ceffa152a681fae3c"} Oct 30 19:15:03.987515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.987492 4789 generic.go:356] "Generic (PLEG): container finished" podID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerID="c6d253d48ae40249456e0241116e60d7a895648a5e8a35ffb38387ada55be8dc" exitCode=60 Oct 30 19:15:03.987625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.987600 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerDied","Data":"c6d253d48ae40249456e0241116e60d7a895648a5e8a35ffb38387ada55be8dc"} Oct 30 19:15:03.987885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:03.987866 4789 scope.go:122] "RemoveContainer" containerID="c6d253d48ae40249456e0241116e60d7a895648a5e8a35ffb38387ada55be8dc" Oct 30 19:15:04.081729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:04.075814 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx"] Oct 30 19:15:04.085377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:04.085343 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f067dc6-pttcx"] Oct 30 19:15:05.023802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.016133 4789 generic.go:356] "Generic (PLEG): container finished" podID="fb696126-0e4b-489d-903c-191c12bcf922" containerID="315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2" exitCode=60 Oct 30 19:15:05.023802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.016293 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" event={"ID":"fb696126-0e4b-489d-903c-191c12bcf922","Type":"ContainerDied","Data":"315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2"} Oct 30 19:15:05.023802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.016364 4789 scope.go:122] "RemoveContainer" containerID="88ae3ebeaa32546897f41cd6e59e5ce727ddddb6b6723139804bd05ab5b360b3" Oct 30 19:15:05.023802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.016890 4789 scope.go:122] "RemoveContainer" containerID="315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2" Oct 30 19:15:05.023802 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.017286 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q_odh-ai-gateway-infra(fb696126-0e4b-489d-903c-191c12bcf922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" podUID="fb696126-0e4b-489d-903c-191c12bcf922" Oct 30 19:15:05.054684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.053650 4789 generic.go:356] "Generic (PLEG): container finished" podID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerID="025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a" exitCode=60 Oct 30 19:15:05.054684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.053874 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerDied","Data":"025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a"} Oct 30 19:15:05.054684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.054324 4789 scope.go:122] "RemoveContainer" containerID="025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a" Oct 30 19:15:05.054684 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.054620 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667_odh-ai-gateway-infra(4aed0b86-b96a-4b78-b01b-691d3451e61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" Oct 30 19:15:05.058658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.057882 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerID="ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209" exitCode=60 Oct 30 19:15:05.058658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.057960 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerDied","Data":"ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209"} Oct 30 19:15:05.058658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.058338 4789 scope.go:122] "RemoveContainer" containerID="ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209" Oct 30 19:15:05.058658 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.058593 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db_odh-ai-gateway-infra(dd89999b-7717-4ed8-a91e-4eb905c012bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" Oct 30 19:15:05.062500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.061407 4789 generic.go:356] "Generic (PLEG): container finished" podID="3c7db509-565e-4310-808c-06420c73cd0b" containerID="7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f" exitCode=60 Oct 30 19:15:05.062500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.061496 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerDied","Data":"7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f"} Oct 30 19:15:05.062500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.062076 4789 scope.go:122] "RemoveContainer" containerID="7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f" Oct 30 19:15:05.062500 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.062352 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x_odh-ai-gateway-infra(3c7db509-565e-4310-808c-06420c73cd0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" podUID="3c7db509-565e-4310-808c-06420c73cd0b" Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.064194 4789 generic.go:356] "Generic (PLEG): container finished" podID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerID="48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb" exitCode=60 Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.064349 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" event={"ID":"b5bc77e1-ff36-4f87-9f36-b8174fce0171","Type":"ContainerDied","Data":"48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb"} Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.064863 4789 scope.go:122] "RemoveContainer" containerID="48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb" Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.065095 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk_odh-ai-gateway-infra(b5bc77e1-ff36-4f87-9f36-b8174fce0171)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.068052 4789 generic.go:356] "Generic (PLEG): container finished" podID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerID="b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d" exitCode=60 Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.068145 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerDied","Data":"b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d"} Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.068439 4789 scope.go:122] "RemoveContainer" containerID="b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d" Oct 30 19:15:05.068815 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:05.068652 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l_odh-ai-gateway-infra(4258318d-b0c1-4dee-a808-719d06c12aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" Oct 30 19:15:05.070003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.069974 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerStarted","Data":"c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c"} Oct 30 19:15:05.080240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.080196 4789 scope.go:122] "RemoveContainer" containerID="1e3f4a1d2486c6a6b2a07c5ae62c74645e148c44dccb6f6f00968c6f07b7652a" Oct 30 19:15:05.090118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.090097 4789 scope.go:122] "RemoveContainer" containerID="22debd74ca1aea10645d9f81620ad5e21c1d11aaa5782eb296cac2f8c877a777" Oct 30 19:15:05.165472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.163988 4789 scope.go:122] "RemoveContainer" containerID="ddbad1b6a3eba27ac58f04f288fdfa2e41b7b7e9ed7b4ceff1dec7962b15f3f4" Oct 30 19:15:05.182228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.182190 4789 scope.go:122] "RemoveContainer" containerID="5be44879a922cb8b4ab0e7def447556cdd70293f318bd5927378f6891cfa7740" Oct 30 19:15:05.276430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.266550 4789 scope.go:122] "RemoveContainer" containerID="eeba5a9d992a53ea7929f3a6c66da508f5e79f0c88bdd02efb914dc2b7ea617b" Oct 30 19:15:05.288659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.288630 4789 scope.go:122] "RemoveContainer" containerID="6ba96477570f3534279424647fc990c8a03707ca013c4f7ceffa152a681fae3c" Oct 30 19:15:05.670316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:05.670283 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af3be7e-b45a-4bf1-94f8-589de1c92175" path="/var/lib/kubelet/pods/8af3be7e-b45a-4bf1-94f8-589de1c92175/volumes" Oct 30 19:15:06.080015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.079990 4789 scope.go:122] "RemoveContainer" containerID="b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d" Oct 30 19:15:06.080728 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.080676 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l_odh-ai-gateway-infra(4258318d-b0c1-4dee-a808-719d06c12aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" Oct 30 19:15:06.081366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.081341 4789 generic.go:356] "Generic (PLEG): container finished" podID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerID="c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c" exitCode=60 Oct 30 19:15:06.081446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.081418 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerDied","Data":"c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c"} Oct 30 19:15:06.081536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.081484 4789 scope.go:122] "RemoveContainer" containerID="c6d253d48ae40249456e0241116e60d7a895648a5e8a35ffb38387ada55be8dc" Oct 30 19:15:06.081655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.081636 4789 scope.go:122] "RemoveContainer" containerID="c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c" Oct 30 19:15:06.081886 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.081865 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc_odh-ai-gateway-infra(b23b3e98-f0cc-455d-a226-30984fa249d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" Oct 30 19:15:06.083383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.083369 4789 scope.go:122] "RemoveContainer" containerID="315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2" Oct 30 19:15:06.083622 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.083606 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q_odh-ai-gateway-infra(fb696126-0e4b-489d-903c-191c12bcf922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" podUID="fb696126-0e4b-489d-903c-191c12bcf922" Oct 30 19:15:06.085838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.085815 4789 scope.go:122] "RemoveContainer" containerID="025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a" Oct 30 19:15:06.086469 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.086453 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667_odh-ai-gateway-infra(4aed0b86-b96a-4b78-b01b-691d3451e61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" Oct 30 19:15:06.087381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.087364 4789 scope.go:122] "RemoveContainer" containerID="ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209" Oct 30 19:15:06.087562 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.087543 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db_odh-ai-gateway-infra(dd89999b-7717-4ed8-a91e-4eb905c012bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" Oct 30 19:15:06.089032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.089016 4789 scope.go:122] "RemoveContainer" containerID="7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f" Oct 30 19:15:06.089233 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.089192 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x_odh-ai-gateway-infra(3c7db509-565e-4310-808c-06420c73cd0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" podUID="3c7db509-565e-4310-808c-06420c73cd0b" Oct 30 19:15:06.091043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:06.091024 4789 scope.go:122] "RemoveContainer" containerID="48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb" Oct 30 19:15:06.118233 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:06.091275 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk_odh-ai-gateway-infra(b5bc77e1-ff36-4f87-9f36-b8174fce0171)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" Oct 30 19:15:07.113231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:07.112275 4789 scope.go:122] "RemoveContainer" containerID="c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c" Oct 30 19:15:07.113231 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:07.112755 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc_odh-ai-gateway-infra(b23b3e98-f0cc-455d-a226-30984fa249d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" Oct 30 19:15:09.958614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:09.958579 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7402-9225d2"] Oct 30 19:15:09.965474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:09.965449 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:10.014927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.014891 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7402-9225d2"] Oct 30 19:15:10.044285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.044238 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dnm\" (UniqueName: \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\") pod \"test-curl-7402-9225d2\" (UID: \"3e958d4e-d4bb-4827-a0a6-755ada295139\") " pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:10.054073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.054041 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7hnn9\"" Oct 30 19:15:10.145825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.145787 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dnm\" (UniqueName: \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\") pod \"test-curl-7402-9225d2\" (UID: \"3e958d4e-d4bb-4827-a0a6-755ada295139\") " pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:10.204887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.204853 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dnm\" (UniqueName: \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\") pod \"test-curl-7402-9225d2\" (UID: \"3e958d4e-d4bb-4827-a0a6-755ada295139\") " pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:10.280033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.279932 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:10.681205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.681168 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7402-9225d2"] Oct 30 19:15:10.685308 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:10.685283 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e958d4e_d4bb_4827_a0a6_755ada295139.slice/crio-a861d2a02ff268c4f008cdd35171fdb507148e645e27ec8d99aa5d60f70e665a WatchSource:0}: Error finding container a861d2a02ff268c4f008cdd35171fdb507148e645e27ec8d99aa5d60f70e665a: Status 404 returned error can't find the container with id a861d2a02ff268c4f008cdd35171fdb507148e645e27ec8d99aa5d60f70e665a Oct 30 19:15:10.687153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:10.687137 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:15:11.010871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.010837 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:11.011365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.010889 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:11.153900 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.153864 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7402-9225d2" event={"ID":"3e958d4e-d4bb-4827-a0a6-755ada295139","Type":"ContainerStarted","Data":"a861d2a02ff268c4f008cdd35171fdb507148e645e27ec8d99aa5d60f70e665a"} Oct 30 19:15:11.155541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.155516 4789 generic.go:356] "Generic (PLEG): container finished" podID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerID="d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937" exitCode=0 Oct 30 19:15:11.155665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.155589 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerDied","Data":"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937"} Oct 30 19:15:11.156101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.156083 4789 scope.go:122] "RemoveContainer" containerID="d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937" Oct 30 19:15:11.161494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.161468 4789 generic.go:356] "Generic (PLEG): container finished" podID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerID="fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8" exitCode=0 Oct 30 19:15:11.161615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.161542 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerDied","Data":"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8"} Oct 30 19:15:11.161934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.161903 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:11.161986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.161949 4789 scope.go:122] "RemoveContainer" containerID="fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8" Oct 30 19:15:11.162021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.161990 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:11.162054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.162034 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:11.163265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.163240 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:11.163343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.163301 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:11.163343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:11.163331 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:12.173929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.173890 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerStarted","Data":"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1"} Oct 30 19:15:12.176094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.176072 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerStarted","Data":"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e"} Oct 30 19:15:12.208305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.207708 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:15:12.208305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.207836 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:12.208305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.207900 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:12.220255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.212178 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:15:12.220255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.212364 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:12.220255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:12.212414 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:13.185870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.185813 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7402-9225d2" event={"ID":"3e958d4e-d4bb-4827-a0a6-755ada295139","Type":"ContainerStarted","Data":"40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac"} Oct 30 19:15:13.216233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.215780 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:13.216233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.215839 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:13.216864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.216840 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:13.217055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.217036 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:13.322234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:13.321225 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7402-9225d2" podStartSLOduration=2.840214176 podStartE2EDuration="4.321183302s" totalImagesPullingTime="1.480969126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:13.312543361 +0000 UTC m=+1749.814453848" watchObservedRunningTime="2026-10-30 19:15:13.321183302 +0000 UTC m=+1749.823093790" Oct 30 19:15:14.232233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:14.216688 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:14.232233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:14.216782 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:14.232233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:14.217419 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:14.232233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:14.217455 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:15.073010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.072971 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq"] Oct 30 19:15:15.130236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.130046 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a294f63-jmwqq"] Oct 30 19:15:15.213664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.196576 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:15.213664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.196651 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:15.237234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.233962 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:15.248253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.247272 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:15.661813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.661781 4789 scope.go:122] "RemoveContainer" containerID="ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209" Oct 30 19:15:15.662141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.662122 4789 scope.go:122] "RemoveContainer" containerID="48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb" Oct 30 19:15:15.672087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:15.672061 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f7e488-4b0c-45fb-accf-e4f21eab6e7c" path="/var/lib/kubelet/pods/a1f7e488-4b0c-45fb-accf-e4f21eab6e7c/volumes" Oct 30 19:15:16.201604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.201492 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:16.201604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.201547 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:16.214296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.212630 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:16.214296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.212685 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:16.256236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.256157 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerStarted","Data":"0384ee87405739efb990967334a79316727dea491ab779fdc543c3b3ce9a3688"} Oct 30 19:15:16.376102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:16.376048 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" podStartSLOduration=16.376034468 podStartE2EDuration="16.376034468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.37388422 +0000 UTC m=+1752.875794706" watchObservedRunningTime="2026-10-30 19:15:16.376034468 +0000 UTC m=+1752.877944953" Oct 30 19:15:17.198235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.195648 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 19:15:17.198235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.195722 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 19:15:17.215675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.215640 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:15:17.215941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.215912 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 30 19:15:17.270403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.270368 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerID="0384ee87405739efb990967334a79316727dea491ab779fdc543c3b3ce9a3688" exitCode=60 Oct 30 19:15:17.270899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.270447 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerDied","Data":"0384ee87405739efb990967334a79316727dea491ab779fdc543c3b3ce9a3688"} Oct 30 19:15:17.270899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.270518 4789 scope.go:122] "RemoveContainer" containerID="ebc3a1549853fefe851e4043208f54920a166a0dd38fd5795abeb170edcef209" Oct 30 19:15:17.271009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.270997 4789 scope.go:122] "RemoveContainer" containerID="0384ee87405739efb990967334a79316727dea491ab779fdc543c3b3ce9a3688" Oct 30 19:15:17.271405 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:17.271378 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db_odh-ai-gateway-infra(dd89999b-7717-4ed8-a91e-4eb905c012bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" Oct 30 19:15:17.272628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.272601 4789 generic.go:356] "Generic (PLEG): container finished" podID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerID="2ebb85320df3ffa82bbc3d6ebf43f8af3c8801d132764e565e78e0ceb69d3c4e" exitCode=60 Oct 30 19:15:17.272730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.272661 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" event={"ID":"b5bc77e1-ff36-4f87-9f36-b8174fce0171","Type":"ContainerDied","Data":"2ebb85320df3ffa82bbc3d6ebf43f8af3c8801d132764e565e78e0ceb69d3c4e"} Oct 30 19:15:17.273036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.273021 4789 scope.go:122] "RemoveContainer" containerID="2ebb85320df3ffa82bbc3d6ebf43f8af3c8801d132764e565e78e0ceb69d3c4e" Oct 30 19:15:17.273312 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:17.273294 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk_odh-ai-gateway-infra(b5bc77e1-ff36-4f87-9f36-b8174fce0171)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" Oct 30 19:15:17.285204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.285172 4789 scope.go:122] "RemoveContainer" containerID="48c381d95f00e57b2c487052cfcb9ffda696f4f95bee84c8bccbb86a1707ebcb" Oct 30 19:15:17.413301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.413241 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:17.672187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.672099 4789 scope.go:122] "RemoveContainer" containerID="c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c" Oct 30 19:15:17.672396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:17.672277 4789 scope.go:122] "RemoveContainer" containerID="b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d" Oct 30 19:15:18.196946 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:18.196907 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:15:18.197803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.197780 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:15:18.198244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.198229 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:15:18.198913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.198895 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:15:18.216749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.216725 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:15:18.217031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.217006 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:15:18.217692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.217674 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:15:18.282265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.282168 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerStarted","Data":"840d409e36623c3c01edaf6f8b0ce63635e534d003b52be159d1c4341e601652"} Oct 30 19:15:18.284094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.284073 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerStarted","Data":"bde222e6b5d43baa1d1ec9c6d6d38d80582796eb19ad95d7e897c20d191e02a6"} Oct 30 19:15:18.470136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.470042 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:18.506003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.505967 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:18.564159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.564073 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\") pod \"dd89999b-7717-4ed8-a91e-4eb905c012bd\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " Oct 30 19:15:18.564159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.564148 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\" (UniqueName: \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\") pod \"dd89999b-7717-4ed8-a91e-4eb905c012bd\" (UID: \"dd89999b-7717-4ed8-a91e-4eb905c012bd\") " Oct 30 19:15:18.564601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.564574 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca" pod "dd89999b-7717-4ed8-a91e-4eb905c012bd" (UID: "dd89999b-7717-4ed8-a91e-4eb905c012bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.564949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.564927 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd89999b-7717-4ed8-a91e-4eb905c012bd-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.566737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.566715 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb" pod "dd89999b-7717-4ed8-a91e-4eb905c012bd" (UID: "dd89999b-7717-4ed8-a91e-4eb905c012bd"). InnerVolumeSpecName "kube-api-access-76nnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.661695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.661660 4789 scope.go:122] "RemoveContainer" containerID="025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a" Oct 30 19:15:18.662656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.662629 4789 scope.go:122] "RemoveContainer" containerID="7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f" Oct 30 19:15:18.667253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.667058 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nnb\" (UniqueName: \"kubernetes.io/projected/dd89999b-7717-4ed8-a91e-4eb905c012bd-kube-api-access-76nnb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.667363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.667282 4789 scope.go:122] "RemoveContainer" containerID="315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2" Oct 30 19:15:18.773969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.773921 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" podStartSLOduration=18.773907863 podStartE2EDuration="18.773907863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:18.770183745 +0000 UTC m=+1755.272094234" watchObservedRunningTime="2026-10-30 19:15:18.773907863 +0000 UTC m=+1755.275818349" Oct 30 19:15:18.791351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.791324 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:18.869737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.869667 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\") pod \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " Oct 30 19:15:18.869866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.869763 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\" (UniqueName: \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\") pod \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\" (UID: \"b5bc77e1-ff36-4f87-9f36-b8174fce0171\") " Oct 30 19:15:18.870062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.870037 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca" pod "b5bc77e1-ff36-4f87-9f36-b8174fce0171" (UID: "b5bc77e1-ff36-4f87-9f36-b8174fce0171"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.870281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.870260 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5bc77e1-ff36-4f87-9f36-b8174fce0171-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.871815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.871795 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s" pod "b5bc77e1-ff36-4f87-9f36-b8174fce0171" (UID: "b5bc77e1-ff36-4f87-9f36-b8174fce0171"). InnerVolumeSpecName "kube-api-access-ggb5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.971681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:18.971648 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb5s\" (UniqueName: \"kubernetes.io/projected/b5bc77e1-ff36-4f87-9f36-b8174fce0171-kube-api-access-ggb5s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:19.298724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.298695 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerStarted","Data":"dc79c129cbf626cc78df65c1ab0bc3d0ffbb7c5351f1c96af3f8cf702ab12465"} Oct 30 19:15:19.300349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.300320 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" event={"ID":"dd89999b-7717-4ed8-a91e-4eb905c012bd","Type":"ContainerDied","Data":"2a1b754ddb0f6e9d0efa63771ca34576e62d6282aedd2cc220a5776a2f8d446c"} Oct 30 19:15:19.300471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.300382 4789 scope.go:122] "RemoveContainer" containerID="0384ee87405739efb990967334a79316727dea491ab779fdc543c3b3ce9a3688" Oct 30 19:15:19.300535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.300491 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db" Oct 30 19:15:19.303682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.303657 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerStarted","Data":"9da97d6e6b2a5b175fdd9ea88412022886917bf8aa18e1f7f14b3c2a6b2bae2c"} Oct 30 19:15:19.305611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.305589 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" Oct 30 19:15:19.305712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.305625 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk" event={"ID":"b5bc77e1-ff36-4f87-9f36-b8174fce0171","Type":"ContainerDied","Data":"a6e83a29318d0b65b2a2559eb5f071071376080b3ece5549104371fb6d5960a4"} Oct 30 19:15:19.307694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.307673 4789 generic.go:356] "Generic (PLEG): container finished" podID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerID="840d409e36623c3c01edaf6f8b0ce63635e534d003b52be159d1c4341e601652" exitCode=60 Oct 30 19:15:19.307841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.307743 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerDied","Data":"840d409e36623c3c01edaf6f8b0ce63635e534d003b52be159d1c4341e601652"} Oct 30 19:15:19.308503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.308143 4789 scope.go:122] "RemoveContainer" containerID="840d409e36623c3c01edaf6f8b0ce63635e534d003b52be159d1c4341e601652" Oct 30 19:15:19.308503 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:19.308419 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l_odh-ai-gateway-infra(4258318d-b0c1-4dee-a808-719d06c12aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" Oct 30 19:15:19.309926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.309907 4789 generic.go:356] "Generic (PLEG): container finished" podID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerID="bde222e6b5d43baa1d1ec9c6d6d38d80582796eb19ad95d7e897c20d191e02a6" exitCode=60 Oct 30 19:15:19.310018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.309984 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerDied","Data":"bde222e6b5d43baa1d1ec9c6d6d38d80582796eb19ad95d7e897c20d191e02a6"} Oct 30 19:15:19.310636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.310619 4789 scope.go:122] "RemoveContainer" containerID="bde222e6b5d43baa1d1ec9c6d6d38d80582796eb19ad95d7e897c20d191e02a6" Oct 30 19:15:19.315996 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:19.310814 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc_odh-ai-gateway-infra(b23b3e98-f0cc-455d-a226-30984fa249d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" Oct 30 19:15:19.318807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.318786 4789 scope.go:122] "RemoveContainer" containerID="2ebb85320df3ffa82bbc3d6ebf43f8af3c8801d132764e565e78e0ceb69d3c4e" Oct 30 19:15:19.329286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.329266 4789 scope.go:122] "RemoveContainer" containerID="b517f513f63166cda5cf71a4b5c8194fbd16fdbf8617ee9677bbd0bcfbf3400d" Oct 30 19:15:19.340894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.340878 4789 scope.go:122] "RemoveContainer" containerID="c157a47c7dd45650159a406eae0d9448676b4a4f13ef51329b621d2a77beb36c" Oct 30 19:15:19.411708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.411640 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" podStartSLOduration=19.411626551 podStartE2EDuration="19.411626551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.40830012 +0000 UTC m=+1755.910210605" watchObservedRunningTime="2026-10-30 19:15:19.411626551 +0000 UTC m=+1755.913537053" Oct 30 19:15:19.560297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.560197 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:19.624323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.624291 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:19.633170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.633138 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889795-fhvtk"] Oct 30 19:15:19.673078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.673033 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" path="/var/lib/kubelet/pods/b5bc77e1-ff36-4f87-9f36-b8174fce0171/volumes" Oct 30 19:15:19.736932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.736887 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" podStartSLOduration=19.736873018 podStartE2EDuration="19.736873018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.736195671 +0000 UTC m=+1756.238106157" watchObservedRunningTime="2026-10-30 19:15:19.736873018 +0000 UTC m=+1756.238783504" Oct 30 19:15:19.783005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.782975 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:19.954176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.954148 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:19.958657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:19.958632 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f1b38f-29889795-2p8db"] Oct 30 19:15:20.053790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.053738 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr"] Oct 30 19:15:20.057957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.057931 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152d03dd-b92wr"] Oct 30 19:15:20.322891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.322854 4789 generic.go:356] "Generic (PLEG): container finished" podID="3c7db509-565e-4310-808c-06420c73cd0b" containerID="9da97d6e6b2a5b175fdd9ea88412022886917bf8aa18e1f7f14b3c2a6b2bae2c" exitCode=60 Oct 30 19:15:20.323476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.322927 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerDied","Data":"9da97d6e6b2a5b175fdd9ea88412022886917bf8aa18e1f7f14b3c2a6b2bae2c"} Oct 30 19:15:20.323476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.323000 4789 scope.go:122] "RemoveContainer" containerID="7271f098f1250d108eb3c346b19cb3026b73c940f6374f8df5d60680c1b9c32f" Oct 30 19:15:20.323579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.323563 4789 scope.go:122] "RemoveContainer" containerID="9da97d6e6b2a5b175fdd9ea88412022886917bf8aa18e1f7f14b3c2a6b2bae2c" Oct 30 19:15:20.323937 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:20.323914 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x_odh-ai-gateway-infra(3c7db509-565e-4310-808c-06420c73cd0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" podUID="3c7db509-565e-4310-808c-06420c73cd0b" Oct 30 19:15:20.333828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.333800 4789 generic.go:356] "Generic (PLEG): container finished" podID="fb696126-0e4b-489d-903c-191c12bcf922" containerID="a33d7aa2e243f408e3e39e28c05a6550a61453f455de21a7f4a511ed384ece17" exitCode=60 Oct 30 19:15:20.333949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.333930 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" event={"ID":"fb696126-0e4b-489d-903c-191c12bcf922","Type":"ContainerDied","Data":"a33d7aa2e243f408e3e39e28c05a6550a61453f455de21a7f4a511ed384ece17"} Oct 30 19:15:20.334389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.334372 4789 scope.go:122] "RemoveContainer" containerID="a33d7aa2e243f408e3e39e28c05a6550a61453f455de21a7f4a511ed384ece17" Oct 30 19:15:20.334758 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:20.334716 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q_odh-ai-gateway-infra(fb696126-0e4b-489d-903c-191c12bcf922)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" podUID="fb696126-0e4b-489d-903c-191c12bcf922" Oct 30 19:15:20.336975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.336947 4789 generic.go:356] "Generic (PLEG): container finished" podID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerID="dc79c129cbf626cc78df65c1ab0bc3d0ffbb7c5351f1c96af3f8cf702ab12465" exitCode=60 Oct 30 19:15:20.337096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.337037 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerDied","Data":"dc79c129cbf626cc78df65c1ab0bc3d0ffbb7c5351f1c96af3f8cf702ab12465"} Oct 30 19:15:20.337447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.337430 4789 scope.go:122] "RemoveContainer" containerID="dc79c129cbf626cc78df65c1ab0bc3d0ffbb7c5351f1c96af3f8cf702ab12465" Oct 30 19:15:20.337765 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:20.337743 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667_odh-ai-gateway-infra(4aed0b86-b96a-4b78-b01b-691d3451e61e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" Oct 30 19:15:20.344725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.341576 4789 scope.go:122] "RemoveContainer" containerID="315672c40dd99d8b43118eb78b85e1f432813b355ec9a03b03963cd5dc0ffff2" Oct 30 19:15:20.358422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.358395 4789 scope.go:122] "RemoveContainer" containerID="025df3a7c55baf1af76a34bc499678dcfadd347fbd069f4d0952b8c62008e14a" Oct 30 19:15:20.424924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.424891 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:20.535564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.535536 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:20.571616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.571590 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:20.697640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.697608 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\") pod \"b23b3e98-f0cc-455d-a226-30984fa249d6\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " Oct 30 19:15:20.697801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.697739 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\" (UniqueName: \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\") pod \"4258318d-b0c1-4dee-a808-719d06c12aa7\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " Oct 30 19:15:20.697846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.697803 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\" (UniqueName: \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\") pod \"b23b3e98-f0cc-455d-a226-30984fa249d6\" (UID: \"b23b3e98-f0cc-455d-a226-30984fa249d6\") " Oct 30 19:15:20.697892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.697882 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\") pod \"4258318d-b0c1-4dee-a808-719d06c12aa7\" (UID: \"4258318d-b0c1-4dee-a808-719d06c12aa7\") " Oct 30 19:15:20.698016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.697996 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca" pod "b23b3e98-f0cc-455d-a226-30984fa249d6" (UID: "b23b3e98-f0cc-455d-a226-30984fa249d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.698278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.698254 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca" pod "4258318d-b0c1-4dee-a808-719d06c12aa7" (UID: "4258318d-b0c1-4dee-a808-719d06c12aa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:20.698558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.698540 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4258318d-b0c1-4dee-a808-719d06c12aa7-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.698634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.698560 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b23b3e98-f0cc-455d-a226-30984fa249d6-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.700010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.699979 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7" pod "b23b3e98-f0cc-455d-a226-30984fa249d6" (UID: "b23b3e98-f0cc-455d-a226-30984fa249d6"). InnerVolumeSpecName "kube-api-access-dzdt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.700098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.700041 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk" pod "4258318d-b0c1-4dee-a808-719d06c12aa7" (UID: "4258318d-b0c1-4dee-a808-719d06c12aa7"). InnerVolumeSpecName "kube-api-access-2c6wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:20.749780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.749754 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:20.799607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.799573 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6wk\" (UniqueName: \"kubernetes.io/projected/4258318d-b0c1-4dee-a808-719d06c12aa7-kube-api-access-2c6wk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:20.799607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:20.799605 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdt7\" (UniqueName: \"kubernetes.io/projected/b23b3e98-f0cc-455d-a226-30984fa249d6-kube-api-access-dzdt7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.350253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.349846 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" Oct 30 19:15:21.350253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.349855 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l" event={"ID":"4258318d-b0c1-4dee-a808-719d06c12aa7","Type":"ContainerDied","Data":"a1bcc9e5a69bab8e853a775cf56bf54fc71d76c7fb2c923eac9e6598b6693eaa"} Oct 30 19:15:21.350253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.349938 4789 scope.go:122] "RemoveContainer" containerID="840d409e36623c3c01edaf6f8b0ce63635e534d003b52be159d1c4341e601652" Oct 30 19:15:21.353849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.353473 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" event={"ID":"b23b3e98-f0cc-455d-a226-30984fa249d6","Type":"ContainerDied","Data":"ca2165a1514dd647ed6bb99ddb67a6bd0440587a54872d24294b5d7585a59a6f"} Oct 30 19:15:21.354037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.353999 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc" Oct 30 19:15:21.397989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.397911 4789 scope.go:122] "RemoveContainer" containerID="bde222e6b5d43baa1d1ec9c6d6d38d80582796eb19ad95d7e897c20d191e02a6" Oct 30 19:15:21.624308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.624280 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:21.628446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.628423 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889795-22lbc"] Oct 30 19:15:21.636129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.636103 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:21.665566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.665542 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:21.670626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.670599 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55015f43-727e-4bc7-a4d1-ca2157a507ab" path="/var/lib/kubelet/pods/55015f43-727e-4bc7-a4d1-ca2157a507ab/volumes" Oct 30 19:15:21.671079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.671065 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" path="/var/lib/kubelet/pods/b23b3e98-f0cc-455d-a226-30984fa249d6/volumes" Oct 30 19:15:21.671628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.671611 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" path="/var/lib/kubelet/pods/dd89999b-7717-4ed8-a91e-4eb905c012bd/volumes" Oct 30 19:15:21.673806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.673790 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:21.734364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.734298 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:21.741128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.740974 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-942c5e85-29889795-qls8l"] Oct 30 19:15:21.826611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.826575 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:21.827106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827085 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\" (UniqueName: \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\") pod \"3c7db509-565e-4310-808c-06420c73cd0b\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " Oct 30 19:15:21.827195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827124 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\") pod \"4aed0b86-b96a-4b78-b01b-691d3451e61e\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " Oct 30 19:15:21.827290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827277 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\" (UniqueName: \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\") pod \"4aed0b86-b96a-4b78-b01b-691d3451e61e\" (UID: \"4aed0b86-b96a-4b78-b01b-691d3451e61e\") " Oct 30 19:15:21.827374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827360 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\") pod \"3c7db509-565e-4310-808c-06420c73cd0b\" (UID: \"3c7db509-565e-4310-808c-06420c73cd0b\") " Oct 30 19:15:21.827548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827524 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca" pod "4aed0b86-b96a-4b78-b01b-691d3451e61e" (UID: "4aed0b86-b96a-4b78-b01b-691d3451e61e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.827699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827682 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca" pod "3c7db509-565e-4310-808c-06420c73cd0b" (UID: "3c7db509-565e-4310-808c-06420c73cd0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.827992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827976 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7db509-565e-4310-808c-06420c73cd0b-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.828051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.827997 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed0b86-b96a-4b78-b01b-691d3451e61e-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.829324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.829302 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c" pod "3c7db509-565e-4310-808c-06420c73cd0b" (UID: "3c7db509-565e-4310-808c-06420c73cd0b"). InnerVolumeSpecName "kube-api-access-njb9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.829430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.829407 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5" pod "4aed0b86-b96a-4b78-b01b-691d3451e61e" (UID: "4aed0b86-b96a-4b78-b01b-691d3451e61e"). InnerVolumeSpecName "kube-api-access-r8ms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.928683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.928585 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\" (UniqueName: \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\") pod \"fb696126-0e4b-489d-903c-191c12bcf922\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " Oct 30 19:15:21.928683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.928676 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\") pod \"fb696126-0e4b-489d-903c-191c12bcf922\" (UID: \"fb696126-0e4b-489d-903c-191c12bcf922\") " Oct 30 19:15:21.929043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.929010 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca" pod "fb696126-0e4b-489d-903c-191c12bcf922" (UID: "fb696126-0e4b-489d-903c-191c12bcf922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.929445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.929422 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ms5\" (UniqueName: \"kubernetes.io/projected/4aed0b86-b96a-4b78-b01b-691d3451e61e-kube-api-access-r8ms5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.929548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.929450 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njb9c\" (UniqueName: \"kubernetes.io/projected/3c7db509-565e-4310-808c-06420c73cd0b-kube-api-access-njb9c\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.929548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.929464 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb696126-0e4b-489d-903c-191c12bcf922-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:21.930706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:21.930678 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr" pod "fb696126-0e4b-489d-903c-191c12bcf922" (UID: "fb696126-0e4b-489d-903c-191c12bcf922"). InnerVolumeSpecName "kube-api-access-7pxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:22.031543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.031506 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pxdr\" (UniqueName: \"kubernetes.io/projected/fb696126-0e4b-489d-903c-191c12bcf922-kube-api-access-7pxdr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:22.372707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.372676 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" Oct 30 19:15:22.373333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.372706 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667" event={"ID":"4aed0b86-b96a-4b78-b01b-691d3451e61e","Type":"ContainerDied","Data":"8f11a62a6c419d25a069d25369940f4a1a354ad77d48b7a5e8941cbe39b77a1d"} Oct 30 19:15:22.373333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.372791 4789 scope.go:122] "RemoveContainer" containerID="dc79c129cbf626cc78df65c1ab0bc3d0ffbb7c5351f1c96af3f8cf702ab12465" Oct 30 19:15:22.374571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.374549 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" Oct 30 19:15:22.374699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.374549 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x" event={"ID":"3c7db509-565e-4310-808c-06420c73cd0b","Type":"ContainerDied","Data":"8cbafaab1f548b3dcd1f0f5b88d2e2ffc9f0b9ba876bc3004db30206de05a006"} Oct 30 19:15:22.381615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.381593 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" Oct 30 19:15:22.381729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.381583 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q" event={"ID":"fb696126-0e4b-489d-903c-191c12bcf922","Type":"ContainerDied","Data":"ffd5167c3f6cec35ec16d5eacda04adf643a10a8fbe446851497e5e59b19dde8"} Oct 30 19:15:22.391482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.391464 4789 scope.go:122] "RemoveContainer" containerID="9da97d6e6b2a5b175fdd9ea88412022886917bf8aa18e1f7f14b3c2a6b2bae2c" Oct 30 19:15:22.401008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.400990 4789 scope.go:122] "RemoveContainer" containerID="a33d7aa2e243f408e3e39e28c05a6550a61453f455de21a7f4a511ed384ece17" Oct 30 19:15:22.531304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.531274 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:22.535388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.535366 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889795-q7l4x"] Oct 30 19:15:22.637506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.637409 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:22.641502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.641476 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-c4ca9a-29889795-cw84q"] Oct 30 19:15:22.744044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.744015 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:22.748919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:22.748896 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889795-rz667"] Oct 30 19:15:23.668677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:23.668647 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7db509-565e-4310-808c-06420c73cd0b" path="/var/lib/kubelet/pods/3c7db509-565e-4310-808c-06420c73cd0b/volumes" Oct 30 19:15:23.669065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:23.669050 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" path="/var/lib/kubelet/pods/4258318d-b0c1-4dee-a808-719d06c12aa7/volumes" Oct 30 19:15:23.669498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:23.669483 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" path="/var/lib/kubelet/pods/4aed0b86-b96a-4b78-b01b-691d3451e61e/volumes" Oct 30 19:15:23.670194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:23.670177 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb696126-0e4b-489d-903c-191c12bcf922" path="/var/lib/kubelet/pods/fb696126-0e4b-489d-903c-191c12bcf922/volumes" Oct 30 19:15:24.112580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:24.112546 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c"] Oct 30 19:15:24.156216 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:24.156189 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12505fe0-t9t7c"] Oct 30 19:15:24.949029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:24.948997 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7402-9225d2"] Oct 30 19:15:24.949463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:24.949251 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7402-9225d2" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" containerName="test-curl-7402-9225d2" containerID="cri-o://40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac" gracePeriod=2 Oct 30 19:15:24.953098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:24.953076 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7402-9225d2"] Oct 30 19:15:24.997309 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:24.997263 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7402-9225d2\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:25.669828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:25.669796 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1be968d-8797-4ccb-b0c9-881ced34c51a" path="/var/lib/kubelet/pods/e1be968d-8797-4ccb-b0c9-881ced34c51a/volumes" Oct 30 19:15:26.361637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.361605 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24"] Oct 30 19:15:26.362259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362242 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.362259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362259 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362270 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362275 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362284 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362289 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362301 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362306 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362317 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362322 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362332 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362337 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362345 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362363 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362371 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362376 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362388 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362392 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362401 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362407 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362412 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362416 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362430 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362434 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362440 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" containerName="test-curl-7402-9225d2" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362445 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" containerName="test-curl-7402-9225d2" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362454 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362458 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362466 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362471 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362478 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362482 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362489 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362493 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362502 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362507 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362598 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362606 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" containerName="test-curl-7402-9225d2" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362613 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362618 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362624 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362629 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362637 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362642 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:26.362638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362648 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362655 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362660 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362666 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362673 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362679 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:26.363601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.362684 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:26.367513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.367496 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.381672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.381647 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hhk\" (UniqueName: \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.381779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.381702 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.381933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.381912 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.465597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.465558 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:26.465754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.465651 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:26.465754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.465668 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:15:26.465754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.465741 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:26.472572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.472541 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24"] Oct 30 19:15:26.482991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.482968 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hhk\" (UniqueName: \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.483078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.483018 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.483175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.483159 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.483824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.483805 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.485417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.485399 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.542344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.542315 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hhk\" (UniqueName: \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-2tj24\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:26.681567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:26.681507 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:27.066796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.066764 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24"] Oct 30 19:15:27.071320 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:27.071297 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a316013_3763_4846_82f1_c5417ae9fdea.slice/crio-290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a WatchSource:0}: Error finding container 290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a: Status 404 returned error can't find the container with id 290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a Oct 30 19:15:27.181166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.181142 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:27.191492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.191468 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\" (UniqueName: \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\") pod \"3e958d4e-d4bb-4827-a0a6-755ada295139\" (UID: \"3e958d4e-d4bb-4827-a0a6-755ada295139\") " Oct 30 19:15:27.194056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.194025 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm" pod "3e958d4e-d4bb-4827-a0a6-755ada295139" (UID: "3e958d4e-d4bb-4827-a0a6-755ada295139"). InnerVolumeSpecName "kube-api-access-h4dnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:27.293651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.293628 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dnm\" (UniqueName: \"kubernetes.io/projected/3e958d4e-d4bb-4827-a0a6-755ada295139-kube-api-access-h4dnm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:27.432711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.432680 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" event={"ID":"4a316013-3763-4846-82f1-c5417ae9fdea","Type":"ContainerStarted","Data":"06121218b32947cd370426e23395b315dab9089fc7f1c4290094a903c7bd180c"} Oct 30 19:15:27.433077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.432741 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" event={"ID":"4a316013-3763-4846-82f1-c5417ae9fdea","Type":"ContainerStarted","Data":"290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a"} Oct 30 19:15:27.434253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.434183 4789 generic.go:356] "Generic (PLEG): container finished" podID="3e958d4e-d4bb-4827-a0a6-755ada295139" containerID="40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac" exitCode=137 Oct 30 19:15:27.434455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.434252 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:27.434455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.434271 4789 scope.go:122] "RemoveContainer" containerID="40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac" Oct 30 19:15:27.445433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.445411 4789 scope.go:122] "RemoveContainer" containerID="40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac" Oct 30 19:15:27.445716 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:27.445696 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac\": container with ID starting with 40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac not found: ID does not exist" containerID="40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac" Oct 30 19:15:27.445780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.445723 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac"} err="failed to get container status \"40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac\": rpc error: code = NotFound desc = could not find container \"40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac\": container with ID starting with 40143693e30d1ac6da8e69274345583933e8f788aea2071db5e2055d4c3c4cac not found: ID does not exist" Oct 30 19:15:27.548628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.548581 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" podStartSLOduration=1.548564818 podStartE2EDuration="1.548564818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:27.544868369 +0000 UTC m=+1764.046778852" watchObservedRunningTime="2026-10-30 19:15:27.548564818 +0000 UTC m=+1764.050475305" Oct 30 19:15:27.592559 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:27.592524 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7402-9225d2\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" pod="opendatahub/test-curl-7402-9225d2" Oct 30 19:15:27.669578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:27.669549 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e958d4e-d4bb-4827-a0a6-755ada295139" path="/var/lib/kubelet/pods/3e958d4e-d4bb-4827-a0a6-755ada295139/volumes" Oct 30 19:15:28.445057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:28.445025 4789 generic.go:356] "Generic (PLEG): container finished" podID="4a316013-3763-4846-82f1-c5417ae9fdea" containerID="06121218b32947cd370426e23395b315dab9089fc7f1c4290094a903c7bd180c" exitCode=0 Oct 30 19:15:28.445528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:28.445078 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" event={"ID":"4a316013-3763-4846-82f1-c5417ae9fdea","Type":"ContainerDied","Data":"06121218b32947cd370426e23395b315dab9089fc7f1c4290094a903c7bd180c"} Oct 30 19:15:29.630818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.630791 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:29.735869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.735806 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\" (UniqueName: \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\") pod \"4a316013-3763-4846-82f1-c5417ae9fdea\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " Oct 30 19:15:29.736016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.735970 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\") pod \"4a316013-3763-4846-82f1-c5417ae9fdea\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " Oct 30 19:15:29.736139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.736025 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\") pod \"4a316013-3763-4846-82f1-c5417ae9fdea\" (UID: \"4a316013-3763-4846-82f1-c5417ae9fdea\") " Oct 30 19:15:29.736379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.736342 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca" pod "4a316013-3763-4846-82f1-c5417ae9fdea" (UID: "4a316013-3763-4846-82f1-c5417ae9fdea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:29.736533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.736515 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:29.737953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.737926 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk" pod "4a316013-3763-4846-82f1-c5417ae9fdea" (UID: "4a316013-3763-4846-82f1-c5417ae9fdea"). InnerVolumeSpecName "kube-api-access-z6hhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:29.738032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.738004 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca" pod "4a316013-3763-4846-82f1-c5417ae9fdea" (UID: "4a316013-3763-4846-82f1-c5417ae9fdea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:29.837906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.837880 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a316013-3763-4846-82f1-c5417ae9fdea-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:29.837906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:29.837905 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hhk\" (UniqueName: \"kubernetes.io/projected/4a316013-3763-4846-82f1-c5417ae9fdea-kube-api-access-z6hhk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:30.466997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:30.466954 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" event={"ID":"4a316013-3763-4846-82f1-c5417ae9fdea","Type":"ContainerDied","Data":"290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a"} Oct 30 19:15:30.466997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:30.466975 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24" Oct 30 19:15:30.466997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:30.467003 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290bfaee8ced971c50049f3a756ae6fff5587f85ee8c0a578695f400490c2a1a" Oct 30 19:15:32.055023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:32.054988 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8"] Oct 30 19:15:32.101273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:32.101239 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f404c535-m57b8"] Oct 30 19:15:33.671138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:33.671104 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb42651-c5e6-47fd-90a2-5f46a102e114" path="/var/lib/kubelet/pods/feb42651-c5e6-47fd-90a2-5f46a102e114/volumes" Oct 30 19:15:37.919667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:37.919634 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:15:39.058855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:39.058816 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp"] Oct 30 19:15:39.064261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:39.064236 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed6cc2c-n29qp"] Oct 30 19:15:39.671634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:39.671601 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9430819e-f32c-43c0-a286-36c54ac3d6da" path="/var/lib/kubelet/pods/9430819e-f32c-43c0-a286-36c54ac3d6da/volumes" Oct 30 19:15:42.920967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:42.920929 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" containerID="cri-o://4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a" gracePeriod=25 Oct 30 19:15:43.066540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:43.066507 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h"] Oct 30 19:15:43.112423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:43.112395 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f367b4ee-mnw5h"] Oct 30 19:15:43.668868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:43.668833 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bacce0-beb9-4e1e-853e-622f651e4d7d" path="/var/lib/kubelet/pods/22bacce0-beb9-4e1e-853e-622f651e4d7d/volumes" Oct 30 19:15:44.808930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:44.808895 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:46.092499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.092461 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv"] Oct 30 19:15:46.093354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093330 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:46.093354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093355 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093417 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093426 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093437 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4a316013-3763-4846-82f1-c5417ae9fdea" containerName="revoke-keys" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093444 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a316013-3763-4846-82f1-c5417ae9fdea" containerName="revoke-keys" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093467 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093474 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093490 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:46.093495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093496 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093643 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bc77e1-ff36-4f87-9f36-b8174fce0171" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093656 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd89999b-7717-4ed8-a91e-4eb905c012bd" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093668 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7db509-565e-4310-808c-06420c73cd0b" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093678 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23b3e98-f0cc-455d-a226-30984fa249d6" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093688 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4258318d-b0c1-4dee-a808-719d06c12aa7" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093698 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb696126-0e4b-489d-903c-191c12bcf922" containerName="cleanup" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093708 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a316013-3763-4846-82f1-c5417ae9fdea" containerName="revoke-keys" Oct 30 19:15:46.093741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.093720 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aed0b86-b96a-4b78-b01b-691d3451e61e" containerName="cleanup" Oct 30 19:15:46.099233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.099198 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.145741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.145709 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.145889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.145753 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drphr\" (UniqueName: \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.145931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.145887 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.148707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.148685 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:15:46.154054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.154016 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv"] Oct 30 19:15:46.158552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.157980 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:46.158940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.158561 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:15:46.159223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.159184 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:15:46.247147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.247108 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.247354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.247162 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drphr\" (UniqueName: \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.247354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.247266 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.247799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.247778 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.249705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.249679 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.310311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.310278 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drphr\" (UniqueName: \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\") pod \"maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.414813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.414721 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:46.796993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:46.796958 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv"] Oct 30 19:15:46.803144 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:15:46.803115 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f76f4b1_3adc_46fe_9e96_5c01f67d3737.slice/crio-94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9 WatchSource:0}: Error finding container 94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9: Status 404 returned error can't find the container with id 94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9 Oct 30 19:15:47.607711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:47.607672 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" event={"ID":"9f76f4b1-3adc-46fe-9e96-5c01f67d3737","Type":"ContainerStarted","Data":"ff60d144a4ce0de01b42aea8cab13846220c0a138c488d4611dfbb2f8bd22c2b"} Oct 30 19:15:47.608142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:47.607735 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" event={"ID":"9f76f4b1-3adc-46fe-9e96-5c01f67d3737","Type":"ContainerStarted","Data":"94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9"} Oct 30 19:15:47.756237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:47.756177 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" podStartSLOduration=1.7561567340000002 podStartE2EDuration="1.756156734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:47.72078831 +0000 UTC m=+1784.222698798" watchObservedRunningTime="2026-10-30 19:15:47.756156734 +0000 UTC m=+1784.258067221" Oct 30 19:15:48.197767 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:15:48.197725 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:15:48.617786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:48.617695 4789 generic.go:356] "Generic (PLEG): container finished" podID="9f76f4b1-3adc-46fe-9e96-5c01f67d3737" containerID="ff60d144a4ce0de01b42aea8cab13846220c0a138c488d4611dfbb2f8bd22c2b" exitCode=0 Oct 30 19:15:48.618150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:48.617788 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" event={"ID":"9f76f4b1-3adc-46fe-9e96-5c01f67d3737","Type":"ContainerDied","Data":"ff60d144a4ce0de01b42aea8cab13846220c0a138c488d4611dfbb2f8bd22c2b"} Oct 30 19:15:49.786326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.786303 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:49.897007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.896936 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\" (UniqueName: \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\") pod \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " Oct 30 19:15:49.897007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.896993 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\") pod \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " Oct 30 19:15:49.897241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.897041 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\") pod \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\" (UID: \"9f76f4b1-3adc-46fe-9e96-5c01f67d3737\") " Oct 30 19:15:49.897718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.897457 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca" pod "9f76f4b1-3adc-46fe-9e96-5c01f67d3737" (UID: "9f76f4b1-3adc-46fe-9e96-5c01f67d3737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:49.897718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.897673 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:49.899131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.899103 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca" pod "9f76f4b1-3adc-46fe-9e96-5c01f67d3737" (UID: "9f76f4b1-3adc-46fe-9e96-5c01f67d3737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:49.899449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.899419 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr" pod "9f76f4b1-3adc-46fe-9e96-5c01f67d3737" (UID: "9f76f4b1-3adc-46fe-9e96-5c01f67d3737"). InnerVolumeSpecName "kube-api-access-drphr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:49.998606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.998577 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drphr\" (UniqueName: \"kubernetes.io/projected/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-kube-api-access-drphr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:49.998606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:49.998606 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f76f4b1-3adc-46fe-9e96-5c01f67d3737-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:50.644231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:50.641706 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" event={"ID":"9f76f4b1-3adc-46fe-9e96-5c01f67d3737","Type":"ContainerDied","Data":"94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9"} Oct 30 19:15:50.644231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:50.641772 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a8120767a90bec885f3448490fa94fe851702e2b152380868e8a50a02d2bf9" Oct 30 19:15:50.644231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:50.641851 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv" Oct 30 19:15:54.808354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:54.808315 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:56.592818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.592781 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:15:56.593671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.593649 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9f76f4b1-3adc-46fe-9e96-5c01f67d3737" containerName="revoke-keys" Oct 30 19:15:56.593671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.593671 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f76f4b1-3adc-46fe-9e96-5c01f67d3737" containerName="revoke-keys" Oct 30 19:15:56.593866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.593852 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f76f4b1-3adc-46fe-9e96-5c01f67d3737" containerName="revoke-keys" Oct 30 19:15:56.597608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.597587 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.661067 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.661034 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ec7ea-openshift-default-dockercfg-8tldc\"" Oct 30 19:15:56.700649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.700619 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:15:56.706947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.706910 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc8c\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.706999 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707052 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707087 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707113 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707161 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707355 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707404 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.707527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.707492 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.878290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.866490 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.926901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.926873 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.927714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.927681 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.927999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.927982 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.928970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.928948 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc8c\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.930787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.930766 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.931708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.931689 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.932140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.932124 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.932305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.932290 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.932536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.928664 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.932642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.932063 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.933747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.908133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.933897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.933680 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.934009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.927608 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.955623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.955597 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.964143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.964112 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:56.998018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:56.997986 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc8c\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:57.027137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.025524 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\") pod \"e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:57.216497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.216464 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:57.675488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.675399 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:15:57.686912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.686877 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:57.687023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.686959 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:57.687023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.687007 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:15:57.947171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.947133 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerStarted","Data":"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68"} Oct 30 19:15:57.947483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:57.947457 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerStarted","Data":"a608675810a74fb81fe634d98e091032fa67819dc75457b6f658fb754deeeee1"} Oct 30 19:15:58.063369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:58.063317 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podStartSLOduration=2.06330012 podStartE2EDuration="2.06330012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:58.062086494 +0000 UTC m=+1794.563996994" watchObservedRunningTime="2026-10-30 19:15:58.06330012 +0000 UTC m=+1794.565210607" Oct 30 19:15:58.216806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:58.216716 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:58.222329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:58.222303 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:58.975660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:58.975625 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:15:58.977597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:15:58.977521 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:16:00.053157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.053124 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg"] Oct 30 19:16:00.098915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.098882 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9898de95-jngjg"] Oct 30 19:16:00.499065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.499033 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp"] Oct 30 19:16:00.503838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.503809 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.553907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.553866 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:00.554090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.554016 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:00.554264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.554164 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:16:00.554354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.554268 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:00.559158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.559135 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp"] Oct 30 19:16:00.616038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.616007 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.616342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.616313 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.616437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.616418 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5df\" (UniqueName: \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.717727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.717702 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.717841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.717758 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5df\" (UniqueName: \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.717841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.717830 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.718493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.718468 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.720081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.720058 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.775745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.775693 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5df\" (UniqueName: \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\") pod \"maas-api-revoke-sub-2f2e383b-gcmjp\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:00.819666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:00.819642 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:01.214437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:01.214407 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp"] Oct 30 19:16:01.217963 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:16:01.217927 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd35814f_6da4_438b_b0aa_6c2f738a7b9d.slice/crio-c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4 WatchSource:0}: Error finding container c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4: Status 404 returned error can't find the container with id c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4 Oct 30 19:16:01.671238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:01.671194 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16211ba-eac8-4c04-a0a4-4ccff634ce35" path="/var/lib/kubelet/pods/d16211ba-eac8-4c04-a0a4-4ccff634ce35/volumes" Oct 30 19:16:02.003682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:02.003645 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" event={"ID":"fd35814f-6da4-438b-b0aa-6c2f738a7b9d","Type":"ContainerStarted","Data":"bbb974371f7ae07a0ab172da6393794a79a9edfb3095f95e8eb719715d5cda0c"} Oct 30 19:16:02.003831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:02.003712 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" event={"ID":"fd35814f-6da4-438b-b0aa-6c2f738a7b9d","Type":"ContainerStarted","Data":"c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4"} Oct 30 19:16:02.122191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:02.122138 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" podStartSLOduration=2.122124733 podStartE2EDuration="2.122124733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:02.119379535 +0000 UTC m=+1798.621290017" watchObservedRunningTime="2026-10-30 19:16:02.122124733 +0000 UTC m=+1798.624035218" Oct 30 19:16:03.013280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.013249 4789 generic.go:356] "Generic (PLEG): container finished" podID="fd35814f-6da4-438b-b0aa-6c2f738a7b9d" containerID="bbb974371f7ae07a0ab172da6393794a79a9edfb3095f95e8eb719715d5cda0c" exitCode=0 Oct 30 19:16:03.013617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.013328 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" event={"ID":"fd35814f-6da4-438b-b0aa-6c2f738a7b9d","Type":"ContainerDied","Data":"bbb974371f7ae07a0ab172da6393794a79a9edfb3095f95e8eb719715d5cda0c"} Oct 30 19:16:03.215141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.215118 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:16:03.254448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.254412 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\" (UniqueName: \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\") pod \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " Oct 30 19:16:03.254733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.254694 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\") pod \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\" (UID: \"b33a3ba1-08f1-486c-99a6-bfb48a8f6732\") " Oct 30 19:16:03.255093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.255056 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume" pod "b33a3ba1-08f1-486c-99a6-bfb48a8f6732" (UID: "b33a3ba1-08f1-486c-99a6-bfb48a8f6732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:03.255681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.255651 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:03.256658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.256637 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv" pod "b33a3ba1-08f1-486c-99a6-bfb48a8f6732" (UID: "b33a3ba1-08f1-486c-99a6-bfb48a8f6732"). InnerVolumeSpecName "kube-api-access-bb8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:03.356854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:03.356824 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb8vv\" (UniqueName: \"kubernetes.io/projected/b33a3ba1-08f1-486c-99a6-bfb48a8f6732-kube-api-access-bb8vv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:04.022079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.021996 4789 generic.go:356] "Generic (PLEG): container finished" podID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerID="4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a" exitCode=0 Oct 30 19:16:04.022079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.022051 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" Oct 30 19:16:04.022619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.022092 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" event={"ID":"b33a3ba1-08f1-486c-99a6-bfb48a8f6732","Type":"ContainerDied","Data":"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a"} Oct 30 19:16:04.022619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.022173 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s" event={"ID":"b33a3ba1-08f1-486c-99a6-bfb48a8f6732","Type":"ContainerDied","Data":"d9800aea0689affc27754b1ecb57ffb591c31bbd3cb95e8c1e243d50181d9046"} Oct 30 19:16:04.022619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.022200 4789 scope.go:122] "RemoveContainer" containerID="4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a" Oct 30 19:16:04.035185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.035168 4789 scope.go:122] "RemoveContainer" containerID="4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a" Oct 30 19:16:04.035514 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:16:04.035478 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a\": container with ID starting with 4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a not found: ID does not exist" containerID="4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a" Oct 30 19:16:04.035600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.035541 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a"} err="failed to get container status \"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a\": rpc error: code = NotFound desc = could not find container \"4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a\": container with ID starting with 4d22390625338407d5e4f740a761afc623c668c7ac6c789b803123a14e8c435a not found: ID does not exist" Oct 30 19:16:04.186676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.186647 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:16:04.195587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.195564 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-942c5e85-5949468747-klw9s"] Oct 30 19:16:04.197304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.197283 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:04.210149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.210128 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:16:04.227859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.227840 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:16:04.270858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.270830 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\" (UniqueName: \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\") pod \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " Oct 30 19:16:04.270978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.270966 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\") pod \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " Oct 30 19:16:04.271048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.270996 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\") pod \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\" (UID: \"fd35814f-6da4-438b-b0aa-6c2f738a7b9d\") " Oct 30 19:16:04.271399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.271371 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca" pod "fd35814f-6da4-438b-b0aa-6c2f738a7b9d" (UID: "fd35814f-6da4-438b-b0aa-6c2f738a7b9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:04.271547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.271523 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:04.273538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.273479 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca" pod "fd35814f-6da4-438b-b0aa-6c2f738a7b9d" (UID: "fd35814f-6da4-438b-b0aa-6c2f738a7b9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:04.273634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.273537 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df" pod "fd35814f-6da4-438b-b0aa-6c2f738a7b9d" (UID: "fd35814f-6da4-438b-b0aa-6c2f738a7b9d"). InnerVolumeSpecName "kube-api-access-xl5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:04.373072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.373046 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:04.373072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:04.373072 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5df\" (UniqueName: \"kubernetes.io/projected/fd35814f-6da4-438b-b0aa-6c2f738a7b9d-kube-api-access-xl5df\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:05.033052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.033027 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" Oct 30 19:16:05.033568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.033063 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp" event={"ID":"fd35814f-6da4-438b-b0aa-6c2f738a7b9d","Type":"ContainerDied","Data":"c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4"} Oct 30 19:16:05.033568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.033109 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c230275bd30bd229c36144d81edca6e0299c77144b7e24c3d1d3918d1cb4eee4" Oct 30 19:16:05.238149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.238117 4789 scope.go:122] "RemoveContainer" containerID="c3289c6a806d877c8ba6d50e80cd54dda81981cd1f4f55ea68797ba3119098ff" Oct 30 19:16:05.248311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.248295 4789 scope.go:122] "RemoveContainer" containerID="9aae8eb9651f80c1737e2ab626e0a935c67cde561b98eddb950c59b5080596b6" Oct 30 19:16:05.668742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:05.668670 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" path="/var/lib/kubelet/pods/b33a3ba1-08f1-486c-99a6-bfb48a8f6732/volumes" Oct 30 19:16:07.054016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:07.053978 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns"] Oct 30 19:16:07.096414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:07.096384 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb022906-hrfns"] Oct 30 19:16:07.669845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:07.669813 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc769b1d-710b-4225-974e-699467d4b70c" path="/var/lib/kubelet/pods/bc769b1d-710b-4225-974e-699467d4b70c/volumes" Oct 30 19:16:12.094449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:12.094419 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm"] Oct 30 19:16:12.100761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:12.100732 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed309e5-rpmwm"] Oct 30 19:16:13.669504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:13.669475 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e94802d-f491-4b13-97d2-df3bb5f50c86" path="/var/lib/kubelet/pods/8e94802d-f491-4b13-97d2-df3bb5f50c86/volumes" Oct 30 19:16:14.877577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.877539 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:16:14.879163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879134 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fd35814f-6da4-438b-b0aa-6c2f738a7b9d" containerName="revoke-keys" Oct 30 19:16:14.879163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879161 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd35814f-6da4-438b-b0aa-6c2f738a7b9d" containerName="revoke-keys" Oct 30 19:16:14.879423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879182 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" Oct 30 19:16:14.879423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879192 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" Oct 30 19:16:14.879423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879391 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd35814f-6da4-438b-b0aa-6c2f738a7b9d" containerName="revoke-keys" Oct 30 19:16:14.879423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.879403 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33a3ba1-08f1-486c-99a6-bfb48a8f6732" containerName="payload-pre-processing" Oct 30 19:16:14.886042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.886019 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:14.899015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.898983 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:14.899749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.899717 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvh9l\" (UniqueName: \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:14.915508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.915479 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:16:14.952829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.952800 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ec7ea\"" Oct 30 19:16:14.953010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:14.952895 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ec7ea-dockercfg-bjlw7\"" Oct 30 19:16:15.010627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.010591 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvh9l\" (UniqueName: \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:15.011494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.011463 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:15.018132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.013068 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:15.100111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.100065 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvh9l\" (UniqueName: \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\") pod \"payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:15.202828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.202792 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:15.614430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:15.614361 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:16:15.619597 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:16:15.619567 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b87e22_9b1f_4d89_9910_0a29b886414c.slice/crio-e49ca3c8aec79213868d02b1e99613156cce06ee3073a5fd193a044017179d2d WatchSource:0}: Error finding container e49ca3c8aec79213868d02b1e99613156cce06ee3073a5fd193a044017179d2d: Status 404 returned error can't find the container with id e49ca3c8aec79213868d02b1e99613156cce06ee3073a5fd193a044017179d2d Oct 30 19:16:16.152794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:16.152760 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" event={"ID":"b8b87e22-9b1f-4d89-9910-0a29b886414c","Type":"ContainerStarted","Data":"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6"} Oct 30 19:16:16.153197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:16.152812 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" event={"ID":"b8b87e22-9b1f-4d89-9910-0a29b886414c","Type":"ContainerStarted","Data":"e49ca3c8aec79213868d02b1e99613156cce06ee3073a5fd193a044017179d2d"} Oct 30 19:16:16.153197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:16.152889 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:16.264443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:16.264390 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" podStartSLOduration=2.264377074 podStartE2EDuration="2.264377074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:16.26305568 +0000 UTC m=+1812.764966169" watchObservedRunningTime="2026-10-30 19:16:16.264377074 +0000 UTC m=+1812.766287559" Oct 30 19:16:17.081990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:17.081952 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz"] Oct 30 19:16:17.125667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:17.125638 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b62a5dc-znnlz"] Oct 30 19:16:17.669987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:17.669951 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b66bde-fef1-4ce8-ab86-a42e2d36a48f" path="/var/lib/kubelet/pods/f1b66bde-fef1-4ce8-ab86-a42e2d36a48f/volumes" Oct 30 19:16:18.207422 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:16:18.207381 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:16:23.064748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:23.064713 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld"] Oct 30 19:16:23.111161 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:23.111132 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc44446-pcgld"] Oct 30 19:16:23.670836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:23.670805 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78123e3-c274-49d4-aa21-0a724b74ed75" path="/var/lib/kubelet/pods/e78123e3-c274-49d4-aa21-0a724b74ed75/volumes" Oct 30 19:16:25.415398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.415359 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x"] Oct 30 19:16:25.420583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.420561 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.472535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.472511 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x"] Oct 30 19:16:25.514396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.514356 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:16:25.514544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.514458 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:16:25.514614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.514578 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:16:25.515624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.515604 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:16:25.554389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.554365 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.554504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.554450 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.554646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.554626 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ll9\" (UniqueName: \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.655815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.655789 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ll9\" (UniqueName: \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.655934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.655892 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.655934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.655916 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.656528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.656506 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.658247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.658227 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.713462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.713436 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ll9\" (UniqueName: \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\") pod \"maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:25.736071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:25.735189 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:26.124146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:26.124067 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x"] Oct 30 19:16:26.129973 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:16:26.129947 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569373de_08ec_4b55_ac57_a5997e9ddbfc.slice/crio-5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99 WatchSource:0}: Error finding container 5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99: Status 404 returned error can't find the container with id 5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99 Oct 30 19:16:26.229772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:26.229739 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" event={"ID":"569373de-08ec-4b55-ac57-a5997e9ddbfc","Type":"ContainerStarted","Data":"5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99"} Oct 30 19:16:27.164042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:27.164015 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:16:27.238582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:27.238517 4789 generic.go:356] "Generic (PLEG): container finished" podID="569373de-08ec-4b55-ac57-a5997e9ddbfc" containerID="e27abf0bc51aed910e4b05f4117048f57e86fe61b8473b9b317507d88f9b2531" exitCode=0 Oct 30 19:16:27.238701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:27.238605 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" event={"ID":"569373de-08ec-4b55-ac57-a5997e9ddbfc","Type":"ContainerDied","Data":"e27abf0bc51aed910e4b05f4117048f57e86fe61b8473b9b317507d88f9b2531"} Oct 30 19:16:28.059880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.059846 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn"] Oct 30 19:16:28.105658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.105630 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dc1a9fd-n5fzn"] Oct 30 19:16:28.412555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.412531 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:28.504457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.504429 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\") pod \"569373de-08ec-4b55-ac57-a5997e9ddbfc\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " Oct 30 19:16:28.504615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.504553 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\") pod \"569373de-08ec-4b55-ac57-a5997e9ddbfc\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " Oct 30 19:16:28.504615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.504582 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\" (UniqueName: \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\") pod \"569373de-08ec-4b55-ac57-a5997e9ddbfc\" (UID: \"569373de-08ec-4b55-ac57-a5997e9ddbfc\") " Oct 30 19:16:28.504955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.504916 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca" pod "569373de-08ec-4b55-ac57-a5997e9ddbfc" (UID: "569373de-08ec-4b55-ac57-a5997e9ddbfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:28.505306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.505276 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:28.506636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.506609 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca" pod "569373de-08ec-4b55-ac57-a5997e9ddbfc" (UID: "569373de-08ec-4b55-ac57-a5997e9ddbfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:16:28.506749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.506698 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9" pod "569373de-08ec-4b55-ac57-a5997e9ddbfc" (UID: "569373de-08ec-4b55-ac57-a5997e9ddbfc"). InnerVolumeSpecName "kube-api-access-g7ll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:28.606474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.606439 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569373de-08ec-4b55-ac57-a5997e9ddbfc-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:28.606474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:28.606475 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ll9\" (UniqueName: \"kubernetes.io/projected/569373de-08ec-4b55-ac57-a5997e9ddbfc-kube-api-access-g7ll9\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:29.255898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:29.255872 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" Oct 30 19:16:29.256069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:29.255890 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x" event={"ID":"569373de-08ec-4b55-ac57-a5997e9ddbfc","Type":"ContainerDied","Data":"5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99"} Oct 30 19:16:29.256069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:29.255969 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8c20aed8d30daef934a836e1e906f2422b8cc1eb890492ab9f936e10ee0c99" Oct 30 19:16:29.669485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:29.669412 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0158847-6a0a-4604-87da-a1c015e425ee" path="/var/lib/kubelet/pods/a0158847-6a0a-4604-87da-a1c015e425ee/volumes" Oct 30 19:16:36.842026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:36.841989 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:16:41.065777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.065737 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m"] Oct 30 19:16:41.111893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.111865 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2"] Oct 30 19:16:41.117787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.117758 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2baf277-8sp4m"] Oct 30 19:16:41.123050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.123025 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f717d9e-mxgx2"] Oct 30 19:16:41.669526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.669491 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179a0856-b1df-4c33-83e1-d25c3422f334" path="/var/lib/kubelet/pods/179a0856-b1df-4c33-83e1-d25c3422f334/volumes" Oct 30 19:16:41.670126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.670104 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90349ec0-fada-4176-82e1-c09b23f3d2cb" path="/var/lib/kubelet/pods/90349ec0-fada-4176-82e1-c09b23f3d2cb/volumes" Oct 30 19:16:41.842303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:41.842271 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerName="payload-processing" containerID="cri-o://be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f" gracePeriod=25 Oct 30 19:16:41.872900 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:16:41.872877 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e908c7_b0d4_49c4_bdff_d53a13521253.slice/crio-be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:16:42.136252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.136230 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:16:42.198607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.198581 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\") pod \"64e908c7-b0d4-49c4-bdff-d53a13521253\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " Oct 30 19:16:42.198723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.198623 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\" (UniqueName: \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\") pod \"64e908c7-b0d4-49c4-bdff-d53a13521253\" (UID: \"64e908c7-b0d4-49c4-bdff-d53a13521253\") " Oct 30 19:16:42.199131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.199099 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume" pod "64e908c7-b0d4-49c4-bdff-d53a13521253" (UID: "64e908c7-b0d4-49c4-bdff-d53a13521253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:42.200716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.200688 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b" pod "64e908c7-b0d4-49c4-bdff-d53a13521253" (UID: "64e908c7-b0d4-49c4-bdff-d53a13521253"). InnerVolumeSpecName "kube-api-access-bc85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:42.300877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.300813 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e908c7-b0d4-49c4-bdff-d53a13521253-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:42.300877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.300849 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc85b\" (UniqueName: \"kubernetes.io/projected/64e908c7-b0d4-49c4-bdff-d53a13521253-kube-api-access-bc85b\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:16:42.392097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.392015 4789 generic.go:356] "Generic (PLEG): container finished" podID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerID="be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f" exitCode=0 Oct 30 19:16:42.392097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.392069 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" Oct 30 19:16:42.392296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.392100 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" event={"ID":"64e908c7-b0d4-49c4-bdff-d53a13521253","Type":"ContainerDied","Data":"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f"} Oct 30 19:16:42.392296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.392164 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" event={"ID":"64e908c7-b0d4-49c4-bdff-d53a13521253","Type":"ContainerDied","Data":"02077d7e7f88659d64073da8d4042eadea91de16f0c84708eb612257adfe8f87"} Oct 30 19:16:42.392296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.392192 4789 scope.go:122] "RemoveContainer" containerID="be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f" Oct 30 19:16:42.404966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.404945 4789 scope.go:122] "RemoveContainer" containerID="be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f" Oct 30 19:16:42.405284 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:16:42.405264 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f\": container with ID starting with be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f not found: ID does not exist" containerID="be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f" Oct 30 19:16:42.405355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.405293 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f"} err="failed to get container status \"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f\": rpc error: code = NotFound desc = could not find container \"be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f\": container with ID starting with be2171f3602bdda4be75ed04d6ed5d399872a7d741b1c9c3da4b5147bac7dd5f not found: ID does not exist" Oct 30 19:16:42.549268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.549239 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:16:42.555162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:42.555097 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m"] Oct 30 19:16:43.035321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:43.035285 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-c4ca9a-578cd4448-dk54m" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.109:50052\" within 1s: context deadline exceeded" Oct 30 19:16:43.035492 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:16:43.035371 4789 logging.go:55] [core] [Channel #548 SubChannel #549]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.109:50052", ServerName: "10.132.0.109:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.109:50052: operation was canceled" Oct 30 19:16:43.669591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:43.669561 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" path="/var/lib/kubelet/pods/64e908c7-b0d4-49c4-bdff-d53a13521253/volumes" Oct 30 19:16:47.050249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:47.050219 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz"] Oct 30 19:16:47.095398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:47.095366 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65e42482-jjmlz"] Oct 30 19:16:47.674542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:47.674508 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9" path="/var/lib/kubelet/pods/8ccb8991-e573-41c6-9b4b-f0f7aa51a0e9/volumes" Oct 30 19:16:48.208414 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:16:48.208368 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:16:51.389218 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:51.389169 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv"] Oct 30 19:16:51.440243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:51.438376 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4f4614-sr2lv"] Oct 30 19:16:51.668947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:51.668871 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f76f4b1-3adc-46fe-9e96-5c01f67d3737" path="/var/lib/kubelet/pods/9f76f4b1-3adc-46fe-9e96-5c01f67d3737/volumes" Oct 30 19:16:52.569249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:52.569201 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24"] Oct 30 19:16:52.613835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:52.613800 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-2tj24"] Oct 30 19:16:53.668772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:53.668739 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a316013-3763-4846-82f1-c5417ae9fdea" path="/var/lib/kubelet/pods/4a316013-3763-4846-82f1-c5417ae9fdea/volumes" Oct 30 19:16:54.051974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:54.051944 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5"] Oct 30 19:16:54.097466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:54.097429 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19efe8a8-98lt5"] Oct 30 19:16:55.672861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:55.672830 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6" path="/var/lib/kubelet/pods/8b89a7f8-7870-40b3-a63c-f90aa5e9aaf6/volumes" Oct 30 19:16:59.203496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:59.203463 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:16:59.204085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:16:59.203730 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" containerID="cri-o://1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2" gracePeriod=30 Oct 30 19:17:00.079448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.079409 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2"] Oct 30 19:17:00.080039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080025 4789 cpu_manager.go:414] "containerMap: removing container" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerName="payload-processing" Oct 30 19:17:00.080091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080040 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerName="payload-processing" Oct 30 19:17:00.080091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080063 4789 cpu_manager.go:414] "containerMap: removing container" podUID="569373de-08ec-4b55-ac57-a5997e9ddbfc" containerName="revoke-keys" Oct 30 19:17:00.080091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080068 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569373de-08ec-4b55-ac57-a5997e9ddbfc" containerName="revoke-keys" Oct 30 19:17:00.080195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080186 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e908c7-b0d4-49c4-bdff-d53a13521253" containerName="payload-processing" Oct 30 19:17:00.080245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.080200 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="569373de-08ec-4b55-ac57-a5997e9ddbfc" containerName="revoke-keys" Oct 30 19:17:00.085173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.085153 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.138311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.138277 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2"] Oct 30 19:17:00.162692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.162655 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpgn\" (UniqueName: \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.162921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.162858 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.163009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.162919 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.186565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.186534 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:00.189246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.186804 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:00.189410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.186875 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:00.189500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.186992 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:17:00.266746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.266694 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.267866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.267835 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.269405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.269377 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.269664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.269646 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpgn\" (UniqueName: \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.277622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.277593 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.328118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.328081 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpgn\" (UniqueName: \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\") pod \"maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.400133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.400062 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:00.782915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:00.782882 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2"] Oct 30 19:17:00.788518 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:17:00.788493 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49d9195_becd_4ba9_8e28_43c79dc3e2f7.slice/crio-8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d WatchSource:0}: Error finding container 8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d: Status 404 returned error can't find the container with id 8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d Oct 30 19:17:01.553195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:01.553161 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" event={"ID":"a49d9195-becd-4ba9-8e28-43c79dc3e2f7","Type":"ContainerStarted","Data":"d4bbd2b6c3be66cebcfc844107143c5c44633a107896d93860b3e8fa3b432de0"} Oct 30 19:17:01.553569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:01.553232 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" event={"ID":"a49d9195-becd-4ba9-8e28-43c79dc3e2f7","Type":"ContainerStarted","Data":"8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d"} Oct 30 19:17:01.663723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:01.663670 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" podStartSLOduration=1.663657728 podStartE2EDuration="1.663657728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:01.662066688 +0000 UTC m=+1858.163977176" watchObservedRunningTime="2026-10-30 19:17:01.663657728 +0000 UTC m=+1858.165568214" Oct 30 19:17:02.562947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:02.562907 4789 generic.go:356] "Generic (PLEG): container finished" podID="a49d9195-becd-4ba9-8e28-43c79dc3e2f7" containerID="d4bbd2b6c3be66cebcfc844107143c5c44633a107896d93860b3e8fa3b432de0" exitCode=0 Oct 30 19:17:02.563341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:02.563002 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" event={"ID":"a49d9195-becd-4ba9-8e28-43c79dc3e2f7","Type":"ContainerDied","Data":"d4bbd2b6c3be66cebcfc844107143c5c44633a107896d93860b3e8fa3b432de0"} Oct 30 19:17:03.546563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.546527 4789 patch_prober.go:29] interesting pod/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:17:03.546724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.546587 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:17:03.730078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.730056 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:03.820502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.820443 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\") pod \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " Oct 30 19:17:03.820606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.820563 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\" (UniqueName: \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\") pod \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " Oct 30 19:17:03.820716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.820704 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\") pod \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\" (UID: \"a49d9195-becd-4ba9-8e28-43c79dc3e2f7\") " Oct 30 19:17:03.820765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.820736 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca" pod "a49d9195-becd-4ba9-8e28-43c79dc3e2f7" (UID: "a49d9195-becd-4ba9-8e28-43c79dc3e2f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:03.821036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.821024 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:03.822572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.822545 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca" pod "a49d9195-becd-4ba9-8e28-43c79dc3e2f7" (UID: "a49d9195-becd-4ba9-8e28-43c79dc3e2f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:03.822666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.822605 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn" pod "a49d9195-becd-4ba9-8e28-43c79dc3e2f7" (UID: "a49d9195-becd-4ba9-8e28-43c79dc3e2f7"). InnerVolumeSpecName "kube-api-access-wfpgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:03.922204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.922177 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:03.922204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:03.922203 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpgn\" (UniqueName: \"kubernetes.io/projected/a49d9195-becd-4ba9-8e28-43c79dc3e2f7-kube-api-access-wfpgn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.465311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.465289 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:17:04.530269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530205 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530260 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530298 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530352 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530472 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:04.530509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530480 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530580 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:04.530605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530598 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:04.530697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530679 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530728 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530745 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:04.530788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530758 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530801 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\") pod \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\" (UID: \"8b78bd8d-66ed-4c75-b07f-0836bd52755f\") " Oct 30 19:17:04.530995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.530969 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:04.531399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.531377 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.531502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.531403 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.531502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.531420 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.531502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.531441 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.531502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.531455 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.532635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.532605 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:04.532727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.532668 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:04.532782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.532752 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "kube-api-access-wgcjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:04.532923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.532906 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo" pod "8b78bd8d-66ed-4c75-b07f-0836bd52755f" (UID: "8b78bd8d-66ed-4c75-b07f-0836bd52755f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:04.582438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.582415 4789 generic.go:356] "Generic (PLEG): container finished" podID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerID="1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2" exitCode=0 Oct 30 19:17:04.582561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.582472 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" Oct 30 19:17:04.582561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.582508 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerDied","Data":"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2"} Oct 30 19:17:04.582669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.582584 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt" event={"ID":"8b78bd8d-66ed-4c75-b07f-0836bd52755f","Type":"ContainerDied","Data":"80e0e6b4fb40eaf0c916ec6446af74b0ba77cb7c6635ff5de181c4dc9dd69eb6"} Oct 30 19:17:04.582669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.582611 4789 scope.go:122] "RemoveContainer" containerID="1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2" Oct 30 19:17:04.585703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.585683 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" Oct 30 19:17:04.585792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.585748 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2" event={"ID":"a49d9195-becd-4ba9-8e28-43c79dc3e2f7","Type":"ContainerDied","Data":"8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d"} Oct 30 19:17:04.585861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.585813 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d861f3ca56e2fb1f516d95bb42167035115bc00244e62e20823244de4f6871d" Oct 30 19:17:04.602841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.602818 4789 scope.go:122] "RemoveContainer" containerID="7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01" Oct 30 19:17:04.614317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.614296 4789 scope.go:122] "RemoveContainer" containerID="1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2" Oct 30 19:17:04.614669 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:04.614640 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2\": container with ID starting with 1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2 not found: ID does not exist" containerID="1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2" Oct 30 19:17:04.614762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.614679 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2"} err="failed to get container status \"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2\": rpc error: code = NotFound desc = could not find container \"1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2\": container with ID starting with 1c201729df863f299bb0ef26079cb426ee21eb903fef12b8b0bb6df21b9780f2 not found: ID does not exist" Oct 30 19:17:04.614762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.614704 4789 scope.go:122] "RemoveContainer" containerID="7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01" Oct 30 19:17:04.615080 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:04.615050 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01\": container with ID starting with 7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01 not found: ID does not exist" containerID="7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01" Oct 30 19:17:04.615190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.615082 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01"} err="failed to get container status \"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01\": rpc error: code = NotFound desc = could not find container \"7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01\": container with ID starting with 7b7c5df5635c5111bc894bc933440a62ba07dca010cbed71ea8b4ea4a3fccf01 not found: ID does not exist" Oct 30 19:17:04.632399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.632378 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgcjr\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-kube-api-access-wgcjr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.632399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.632401 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.632532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.632410 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.632532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.632420 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b78bd8d-66ed-4c75-b07f-0836bd52755f-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:04.790877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.790824 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:17:04.795816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:04.795796 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4f4614-openshift-default-7cdfb4fff9-vqfxt"] Oct 30 19:17:05.326966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.326934 4789 scope.go:122] "RemoveContainer" containerID="dc5b2c101b3992cd0298d0c6223273e871c8421a3374ffd2ce4d7c636ba8b842" Oct 30 19:17:05.336924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.336904 4789 scope.go:122] "RemoveContainer" containerID="ed79cb9663ad0d15dbd636149c03f2eb84de4ea30c35d86d817bd94553e47433" Oct 30 19:17:05.345700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.345674 4789 scope.go:122] "RemoveContainer" containerID="68f38e5eb4918196caa915cb8c758e37088efb25ea5771435cef252f52c17ada" Oct 30 19:17:05.354368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.354352 4789 scope.go:122] "RemoveContainer" containerID="dceb9c390b8910b707c51cfdc60d933896dea0781a6a3aa0e3c6f5e18f2b2a88" Oct 30 19:17:05.362798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.362780 4789 scope.go:122] "RemoveContainer" containerID="afd773c6cc746154bdff30a89a8579afc8a7fd49ca32e71e732a2c2b9d98daf4" Oct 30 19:17:05.370942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.370921 4789 scope.go:122] "RemoveContainer" containerID="0a81938a00223d31ac0ed42a598f361edfdd7fc2b0d7c80f3e9d140cabb31b59" Oct 30 19:17:05.379549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.379530 4789 scope.go:122] "RemoveContainer" containerID="6791e6329f332b7d8946e6ebafe1d4914d5e8f55ca3a975fd3d7b036fb38056d" Oct 30 19:17:05.388103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.388087 4789 scope.go:122] "RemoveContainer" containerID="014cb11eb31a1e15973c106cac679c825949494be63da03ab56f3c2c11a66b89" Oct 30 19:17:05.668194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:05.668123 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" path="/var/lib/kubelet/pods/8b78bd8d-66ed-4c75-b07f-0836bd52755f/volumes" Oct 30 19:17:11.475479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:11.475445 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:17:16.475929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.475856 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" podUID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" containerName="payload-processing" containerID="cri-o://e056f10d63a2067ead92a527b44cbcb1d2b0f234c1bf8fc5a936ff948fcb0dab" gracePeriod=25 Oct 30 19:17:16.682932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.682903 4789 generic.go:356] "Generic (PLEG): container finished" podID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" containerID="e056f10d63a2067ead92a527b44cbcb1d2b0f234c1bf8fc5a936ff948fcb0dab" exitCode=0 Oct 30 19:17:16.683094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.682999 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" event={"ID":"e4e8697f-ee19-4a88-825f-5dc40fe8d587","Type":"ContainerDied","Data":"e056f10d63a2067ead92a527b44cbcb1d2b0f234c1bf8fc5a936ff948fcb0dab"} Oct 30 19:17:16.814244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.813479 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:17:16.933122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.933090 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\" (UniqueName: \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\") pod \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " Oct 30 19:17:16.933316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.933299 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\") pod \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\" (UID: \"e4e8697f-ee19-4a88-825f-5dc40fe8d587\") " Oct 30 19:17:16.934063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.933994 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume" pod "e4e8697f-ee19-4a88-825f-5dc40fe8d587" (UID: "e4e8697f-ee19-4a88-825f-5dc40fe8d587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:16.935868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:16.935840 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g" pod "e4e8697f-ee19-4a88-825f-5dc40fe8d587" (UID: "e4e8697f-ee19-4a88-825f-5dc40fe8d587"). InnerVolumeSpecName "kube-api-access-hqv9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:17.035128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.035059 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e8697f-ee19-4a88-825f-5dc40fe8d587-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.035328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.035309 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqv9g\" (UniqueName: \"kubernetes.io/projected/e4e8697f-ee19-4a88-825f-5dc40fe8d587-kube-api-access-hqv9g\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:17.719069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.716612 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" event={"ID":"e4e8697f-ee19-4a88-825f-5dc40fe8d587","Type":"ContainerDied","Data":"2458555b28f2b9840c1c508de53be0b5d9028441151a007bb59753c49f115db9"} Oct 30 19:17:17.719069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.716690 4789 scope.go:122] "RemoveContainer" containerID="e056f10d63a2067ead92a527b44cbcb1d2b0f234c1bf8fc5a936ff948fcb0dab" Oct 30 19:17:17.719069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.716867 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9" Oct 30 19:17:17.876487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.876455 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:17:17.880084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:17.880058 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f1b38f-5d49b8454d-c9nm9"] Oct 30 19:17:18.220186 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:18.220145 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:17:19.669147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:19.669113 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" path="/var/lib/kubelet/pods/e4e8697f-ee19-4a88-825f-5dc40fe8d587/volumes" Oct 30 19:17:25.786144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.786109 4789 generic.go:356] "Generic (PLEG): container finished" podID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerID="972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68" exitCode=0 Oct 30 19:17:25.786635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.786178 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerDied","Data":"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68"} Oct 30 19:17:25.786635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.786611 4789 scope.go:122] "RemoveContainer" containerID="972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68" Oct 30 19:17:25.788076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.788049 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:17:25.788144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.788114 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:17:25.788180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:25.788145 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:17:26.802320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:26.802277 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerStarted","Data":"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0"} Oct 30 19:17:27.222334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:27.217723 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:17:27.226372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:27.226338 4789 patch_prober.go:29] interesting pod/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:17:27.226617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:27.226585 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:17:28.218117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:28.218079 4789 patch_prober.go:29] interesting pod/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:17:28.218540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:28.218145 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:17:28.975822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:28.975790 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:17:29.217714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:29.217676 4789 patch_prober.go:29] interesting pod/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:17:29.217910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:29.217734 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:17:30.217694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:30.217654 4789 patch_prober.go:29] interesting pod/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:17:30.218119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:30.217716 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:17:31.217728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:31.217689 4789 patch_prober.go:29] interesting pod/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:17:31.218358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:31.217753 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:17:32.252124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:32.252087 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:17:32.253572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:32.253546 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:17:33.160686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:33.160656 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:17:33.160917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:33.160882 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" containerID="cri-o://e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af" gracePeriod=30 Oct 30 19:17:33.208617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:33.208589 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:17:33.208808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:33.208787 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" containerID="cri-o://932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f" gracePeriod=30 Oct 30 19:17:37.709262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:37.709229 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x"] Oct 30 19:17:37.755480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:37.755448 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c4ca9a-bp57x"] Oct 30 19:17:38.490092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.490068 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:17:38.523653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.523633 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:17:38.581887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.581863 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.581994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.581922 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.581994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.581956 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582000 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582031 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582091 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582121 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582367 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.582469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582397 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582441 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582503 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582594 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582624 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582673 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.582730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582692 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582755 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582793 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.582837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582821 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\") pod \"0024db27-849f-48c6-bed2-3a60fa7cba15\" (UID: \"0024db27-849f-48c6-bed2-3a60fa7cba15\") " Oct 30 19:17:38.582997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582929 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.583053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.582997 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\") pod \"47c5c660-734c-4584-82aa-7c3a8e6892e4\" (UID: \"47c5c660-734c-4584-82aa-7c3a8e6892e4\") " Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.583328 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.583350 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.583668 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.583957 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584072 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584139 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584341 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584465 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.584676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584485 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:38.584676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.584604 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "kube-api-access-24g5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:38.585096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585070 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585251 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0024db27-849f-48c6-bed2-3a60fa7cba15-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585278 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585294 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585074 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585309 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g5b\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-kube-api-access-24g5b\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585108 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "kube-api-access-gz85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585163 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585336 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585352 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585365 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585379 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585392 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585406 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585413 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:38.585680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585421 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c5c660-734c-4584-82aa-7c3a8e6892e4-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.585997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.585975 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy" pod "47c5c660-734c-4584-82aa-7c3a8e6892e4" (UID: "47c5c660-734c-4584-82aa-7c3a8e6892e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.586291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.586270 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy" pod "0024db27-849f-48c6-bed2-3a60fa7cba15" (UID: "0024db27-849f-48c6-bed2-3a60fa7cba15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:38.687559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687489 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.687559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687523 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.687559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687539 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.687559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687552 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz85d\" (UniqueName: \"kubernetes.io/projected/47c5c660-734c-4584-82aa-7c3a8e6892e4-kube-api-access-gz85d\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.687783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687566 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0024db27-849f-48c6-bed2-3a60fa7cba15-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:38.687783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:38.687579 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c5c660-734c-4584-82aa-7c3a8e6892e4-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:39.008429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.008335 4789 generic.go:356] "Generic (PLEG): container finished" podID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerID="932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f" exitCode=0 Oct 30 19:17:39.008429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.008392 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" Oct 30 19:17:39.008429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.008416 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" event={"ID":"47c5c660-734c-4584-82aa-7c3a8e6892e4","Type":"ContainerDied","Data":"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f"} Oct 30 19:17:39.009015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.008478 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f" event={"ID":"47c5c660-734c-4584-82aa-7c3a8e6892e4","Type":"ContainerDied","Data":"76abd3bc4931996f611f2146cbbaad6b5cdab7ea835f65a49482cdddbb477e9c"} Oct 30 19:17:39.009015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.008510 4789 scope.go:122] "RemoveContainer" containerID="932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f" Oct 30 19:17:39.010048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.010020 4789 generic.go:356] "Generic (PLEG): container finished" podID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerID="e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af" exitCode=0 Oct 30 19:17:39.010173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.010053 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" Oct 30 19:17:39.010173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.010115 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" event={"ID":"0024db27-849f-48c6-bed2-3a60fa7cba15","Type":"ContainerDied","Data":"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af"} Oct 30 19:17:39.010173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.010162 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg" event={"ID":"0024db27-849f-48c6-bed2-3a60fa7cba15","Type":"ContainerDied","Data":"bf6e56158eb7bf056a31f5ee10d4891cf100a929ca4cf475113ca98da44e049e"} Oct 30 19:17:39.020015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.019891 4789 scope.go:122] "RemoveContainer" containerID="932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f" Oct 30 19:17:39.020278 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:39.020259 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f\": container with ID starting with 932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f not found: ID does not exist" containerID="932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f" Oct 30 19:17:39.020350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.020285 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f"} err="failed to get container status \"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f\": rpc error: code = NotFound desc = could not find container \"932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f\": container with ID starting with 932aff1e56ae756ad3cfebf3d77976ae45a58901afde386ddcd2dd120a7cde4f not found: ID does not exist" Oct 30 19:17:39.020350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.020302 4789 scope.go:122] "RemoveContainer" containerID="e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af" Oct 30 19:17:39.029717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.029698 4789 scope.go:122] "RemoveContainer" containerID="e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af" Oct 30 19:17:39.029995 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:39.029977 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af\": container with ID starting with e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af not found: ID does not exist" containerID="e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af" Oct 30 19:17:39.030054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.030004 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af"} err="failed to get container status \"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af\": rpc error: code = NotFound desc = could not find container \"e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af\": container with ID starting with e6f2b7c8cddb14a6a906e09d0c6bdea9733c715ee6d001447e31a6bfd63de9af not found: ID does not exist" Oct 30 19:17:39.166965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.166930 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:17:39.212963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.212932 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-4l7lg"] Oct 30 19:17:39.272760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.272642 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:17:39.279672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.279640 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-mq66f"] Oct 30 19:17:39.668525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.668448 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" path="/var/lib/kubelet/pods/0024db27-849f-48c6-bed2-3a60fa7cba15/volumes" Oct 30 19:17:39.668974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.668960 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" path="/var/lib/kubelet/pods/47c5c660-734c-4584-82aa-7c3a8e6892e4/volumes" Oct 30 19:17:39.669381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:39.669368 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569373de-08ec-4b55-ac57-a5997e9ddbfc" path="/var/lib/kubelet/pods/569373de-08ec-4b55-ac57-a5997e9ddbfc/volumes" Oct 30 19:17:41.052726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.051934 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62"] Oct 30 19:17:41.095719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.095681 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c762ff1-lpd62"] Oct 30 19:17:41.669082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.669049 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6117c25-ecf2-4c99-a733-41e6fe21fd6b" path="/var/lib/kubelet/pods/a6117c25-ecf2-4c99-a733-41e6fe21fd6b/volumes" Oct 30 19:17:41.669529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.669493 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5"] Oct 30 19:17:41.670028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670014 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670030 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670043 4789 cpu_manager.go:414] "containerMap: removing container" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670049 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670063 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670068 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:41.670087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670085 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a49d9195-becd-4ba9-8e28-43c79dc3e2f7" containerName="revoke-keys" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670090 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49d9195-becd-4ba9-8e28-43c79dc3e2f7" containerName="revoke-keys" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670101 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" containerName="payload-processing" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670106 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" containerName="payload-processing" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670205 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670233 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49d9195-becd-4ba9-8e28-43c79dc3e2f7" containerName="revoke-keys" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670242 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e8697f-ee19-4a88-825f-5dc40fe8d587" containerName="payload-processing" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670249 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670260 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0024db27-849f-48c6-bed2-3a60fa7cba15" containerName="istio-proxy" Oct 30 19:17:41.670325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.670266 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c5c660-734c-4584-82aa-7c3a8e6892e4" containerName="istio-proxy" Oct 30 19:17:41.675142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.675124 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.720402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.720368 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5"] Oct 30 19:17:41.735999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.735957 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.736186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.736035 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-454m7\" (UniqueName: \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.736375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.736355 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.762020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.761967 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:41.762020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.761966 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:41.762020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.761967 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:41.762345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.761984 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:17:41.837485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.837449 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.837657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.837569 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.837657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.837617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-454m7\" (UniqueName: \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.838481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.838204 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.840240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.840188 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.896429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.896390 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-454m7\" (UniqueName: \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\") pod \"maas-api-revoke-sub-e8052e80-mxsq5\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:41.989012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:41.988981 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:42.370843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:42.370768 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5"] Oct 30 19:17:42.376930 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:17:42.376890 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef638095_80ad_43ea_ad32_29019f8b6c5a.slice/crio-02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666 WatchSource:0}: Error finding container 02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666: Status 404 returned error can't find the container with id 02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666 Oct 30 19:17:43.042891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:43.042788 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" event={"ID":"ef638095-80ad-43ea-ad32-29019f8b6c5a","Type":"ContainerStarted","Data":"c9fc5aeda50c0edb66c9e768bfa51b9274ae8d73baf552313536688a8f7c852a"} Oct 30 19:17:43.042891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:43.042862 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" event={"ID":"ef638095-80ad-43ea-ad32-29019f8b6c5a","Type":"ContainerStarted","Data":"02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666"} Oct 30 19:17:43.161534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:43.161485 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" podStartSLOduration=2.161470997 podStartE2EDuration="2.161470997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:43.157451639 +0000 UTC m=+1899.659362124" watchObservedRunningTime="2026-10-30 19:17:43.161470997 +0000 UTC m=+1899.663381498" Oct 30 19:17:44.059550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:44.059513 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef638095-80ad-43ea-ad32-29019f8b6c5a" containerID="c9fc5aeda50c0edb66c9e768bfa51b9274ae8d73baf552313536688a8f7c852a" exitCode=0 Oct 30 19:17:44.059979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:44.059594 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" event={"ID":"ef638095-80ad-43ea-ad32-29019f8b6c5a","Type":"ContainerDied","Data":"c9fc5aeda50c0edb66c9e768bfa51b9274ae8d73baf552313536688a8f7c852a"} Oct 30 19:17:45.221032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.221002 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:45.387971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.387884 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\" (UniqueName: \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\") pod \"ef638095-80ad-43ea-ad32-29019f8b6c5a\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " Oct 30 19:17:45.388144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.388007 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\") pod \"ef638095-80ad-43ea-ad32-29019f8b6c5a\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " Oct 30 19:17:45.388144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.388028 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\") pod \"ef638095-80ad-43ea-ad32-29019f8b6c5a\" (UID: \"ef638095-80ad-43ea-ad32-29019f8b6c5a\") " Oct 30 19:17:45.388856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.388823 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca" pod "ef638095-80ad-43ea-ad32-29019f8b6c5a" (UID: "ef638095-80ad-43ea-ad32-29019f8b6c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:45.391008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.390980 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7" pod "ef638095-80ad-43ea-ad32-29019f8b6c5a" (UID: "ef638095-80ad-43ea-ad32-29019f8b6c5a"). InnerVolumeSpecName "kube-api-access-454m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:45.391149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.390986 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca" pod "ef638095-80ad-43ea-ad32-29019f8b6c5a" (UID: "ef638095-80ad-43ea-ad32-29019f8b6c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:45.490161 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.490130 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:45.490161 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.490159 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef638095-80ad-43ea-ad32-29019f8b6c5a-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:45.490369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.490170 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-454m7\" (UniqueName: \"kubernetes.io/projected/ef638095-80ad-43ea-ad32-29019f8b6c5a-kube-api-access-454m7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:45.620194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.619235 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:17:45.620194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:45.619477 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" containerID="cri-o://b34857e992f063d8c6073b6ef3f10f96d37ea8db9f8861b9b0533152e8e5a1f2" gracePeriod=30 Oct 30 19:17:46.078864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.078835 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" Oct 30 19:17:46.079030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.078834 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5" event={"ID":"ef638095-80ad-43ea-ad32-29019f8b6c5a","Type":"ContainerDied","Data":"02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666"} Oct 30 19:17:46.079030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.078958 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02625ab0fe7f05e84881f8a34cb9559a89f535b63492d83ac88bedbd45981666" Oct 30 19:17:46.486304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.486259 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb"] Oct 30 19:17:46.487068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.487037 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef638095-80ad-43ea-ad32-29019f8b6c5a" containerName="revoke-keys" Oct 30 19:17:46.487068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.487060 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef638095-80ad-43ea-ad32-29019f8b6c5a" containerName="revoke-keys" Oct 30 19:17:46.487280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.487143 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:46.487280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.487152 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b78bd8d-66ed-4c75-b07f-0836bd52755f" containerName="istio-proxy" Oct 30 19:17:46.487381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.487324 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef638095-80ad-43ea-ad32-29019f8b6c5a" containerName="revoke-keys" Oct 30 19:17:46.491054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.491031 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.543247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.543222 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:46.543524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.543502 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:17:46.543658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.543571 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:46.543729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.543721 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:46.548185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.548158 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb"] Oct 30 19:17:46.615712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.615677 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.615885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.615721 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gt6\" (UniqueName: \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.615951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.615881 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.664114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.664087 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:17:46.664292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.664271 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" containerID="cri-o://7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec" gracePeriod=30 Oct 30 19:17:46.717564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.717535 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.717761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.717742 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.717845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.717794 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gt6\" (UniqueName: \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.718589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.718553 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.719875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.719856 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.775388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.775325 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gt6\" (UniqueName: \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\") pod \"maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:46.808619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:46.808581 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:47.200232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:47.200179 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb"] Oct 30 19:17:47.206343 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:17:47.206318 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2add2be2_822e_4882_b263_7937439c9514.slice/crio-f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3 WatchSource:0}: Error finding container f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3: Status 404 returned error can't find the container with id f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3 Oct 30 19:17:48.127065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:48.127023 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" event={"ID":"2add2be2-822e-4882-b263-7937439c9514","Type":"ContainerStarted","Data":"8a834b10f6b4257868a12df6f53f9631b9570b42c46939c0a94642a48c5e133a"} Oct 30 19:17:48.127477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:48.127089 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" event={"ID":"2add2be2-822e-4882-b263-7937439c9514","Type":"ContainerStarted","Data":"f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3"} Oct 30 19:17:48.203035 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:48.202991 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:17:48.237937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:48.237880 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" podStartSLOduration=2.237862937 podStartE2EDuration="2.237862937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:48.23761773 +0000 UTC m=+1904.739528220" watchObservedRunningTime="2026-10-30 19:17:48.237862937 +0000 UTC m=+1904.739773425" Oct 30 19:17:49.166230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:49.154329 4789 generic.go:356] "Generic (PLEG): container finished" podID="2add2be2-822e-4882-b263-7937439c9514" containerID="8a834b10f6b4257868a12df6f53f9631b9570b42c46939c0a94642a48c5e133a" exitCode=0 Oct 30 19:17:49.166230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:49.154546 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" event={"ID":"2add2be2-822e-4882-b263-7937439c9514","Type":"ContainerDied","Data":"8a834b10f6b4257868a12df6f53f9631b9570b42c46939c0a94642a48c5e133a"} Oct 30 19:17:50.719120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.719087 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:50.806432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.806399 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\") pod \"2add2be2-822e-4882-b263-7937439c9514\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " Oct 30 19:17:50.808409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.808387 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca" pod "2add2be2-822e-4882-b263-7937439c9514" (UID: "2add2be2-822e-4882-b263-7937439c9514"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:50.907741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.907675 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\") pod \"2add2be2-822e-4882-b263-7937439c9514\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " Oct 30 19:17:50.907850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.907774 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\" (UniqueName: \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\") pod \"2add2be2-822e-4882-b263-7937439c9514\" (UID: \"2add2be2-822e-4882-b263-7937439c9514\") " Oct 30 19:17:50.908057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.908034 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca" pod "2add2be2-822e-4882-b263-7937439c9514" (UID: "2add2be2-822e-4882-b263-7937439c9514"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:50.908334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.908318 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2be2-822e-4882-b263-7937439c9514-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:50.908374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.908337 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2be2-822e-4882-b263-7937439c9514-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:50.909716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:50.909689 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6" pod "2add2be2-822e-4882-b263-7937439c9514" (UID: "2add2be2-822e-4882-b263-7937439c9514"). InnerVolumeSpecName "kube-api-access-x4gt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:51.009650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.009623 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gt6\" (UniqueName: \"kubernetes.io/projected/2add2be2-822e-4882-b263-7937439c9514-kube-api-access-x4gt6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.187014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.186983 4789 generic.go:356] "Generic (PLEG): container finished" podID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerID="b34857e992f063d8c6073b6ef3f10f96d37ea8db9f8861b9b0533152e8e5a1f2" exitCode=0 Oct 30 19:17:51.187162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.187058 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerDied","Data":"b34857e992f063d8c6073b6ef3f10f96d37ea8db9f8861b9b0533152e8e5a1f2"} Oct 30 19:17:51.187162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.187119 4789 scope.go:122] "RemoveContainer" containerID="6381f94f425854393d57beb0ba5d363c96229d486447b124f23d2996d27b804f" Oct 30 19:17:51.188801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.188777 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" Oct 30 19:17:51.188910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.188781 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb" event={"ID":"2add2be2-822e-4882-b263-7937439c9514","Type":"ContainerDied","Data":"f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3"} Oct 30 19:17:51.188910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.188900 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f53f89aab00d2d8d693eb1a8e14eb7d6c94a327b9c0b28d32796a5b6ae2afbe3" Oct 30 19:17:51.299167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.299140 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:17:51.414485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414456 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.414632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414509 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.414632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414548 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.414784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414630 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.414784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414672 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.414784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414716 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414831 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414937 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.414985 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\") pod \"56ad40a6-a845-4446-b6c6-4d87afa3a746\" (UID: \"56ad40a6-a845-4446-b6c6-4d87afa3a746\") " Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415034 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415043 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415261 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:51.415663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415356 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:51.416015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415675 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ad40a6-a845-4446-b6c6-4d87afa3a746-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.416015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415698 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.416015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415712 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.416015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.415748 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.416853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.416814 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:51.417419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.417386 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:51.417890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.417793 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "kube-api-access-k6q56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:51.417976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.417929 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:51.418722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.418693 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy" pod "56ad40a6-a845-4446-b6c6-4d87afa3a746" (UID: "56ad40a6-a845-4446-b6c6-4d87afa3a746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:51.516740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.516685 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.516740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.516709 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6q56\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-kube-api-access-k6q56\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.516740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.516719 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.516740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.516728 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.516740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.516738 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ad40a6-a845-4446-b6c6-4d87afa3a746-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:51.939627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:51.939603 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134257 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134316 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134348 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134417 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134445 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134531 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134558 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134586 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.136289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.134638 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\") pod \"4a115530-4069-411e-9ab5-289553297b1b\" (UID: \"4a115530-4069-411e-9ab5-289553297b1b\") " Oct 30 19:17:52.156229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.142994 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:52.156229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.143463 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:52.156229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.144432 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:52.156229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.144736 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:52.168412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.162961 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:52.168412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.163786 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:52.197251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.194549 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:52.221732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.221568 4789 generic.go:356] "Generic (PLEG): container finished" podID="4a115530-4069-411e-9ab5-289553297b1b" containerID="7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec" exitCode=0 Oct 30 19:17:52.221914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.221837 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" event={"ID":"4a115530-4069-411e-9ab5-289553297b1b","Type":"ContainerDied","Data":"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec"} Oct 30 19:17:52.221982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.221921 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" event={"ID":"4a115530-4069-411e-9ab5-289553297b1b","Type":"ContainerDied","Data":"1f6315a6bee528cbe6f9f98b175cdb48427375215a41480c2cf731207c59df62"} Oct 30 19:17:52.221982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.221951 4789 scope.go:122] "RemoveContainer" containerID="7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec" Oct 30 19:17:52.222130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.222112 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d" Oct 30 19:17:52.225465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.225436 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" event={"ID":"56ad40a6-a845-4446-b6c6-4d87afa3a746","Type":"ContainerDied","Data":"3b4ffb8b43441f80412bd88bba434387ef5ebb6e44fca6c21ea840b6b4272cb3"} Oct 30 19:17:52.225598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.225567 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z" Oct 30 19:17:52.233286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.221116 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "kube-api-access-fskcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:52.233286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.222866 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token" pod "4a115530-4069-411e-9ab5-289553297b1b" (UID: "4a115530-4069-411e-9ab5-289553297b1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236122 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236146 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236160 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a115530-4069-411e-9ab5-289553297b1b-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236175 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskcf\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-kube-api-access-fskcf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236191 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236220 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236233 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a115530-4069-411e-9ab5-289553297b1b-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236245 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a115530-4069-411e-9ab5-289553297b1b-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.238816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.236259 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a115530-4069-411e-9ab5-289553297b1b-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:52.257918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.257897 4789 scope.go:122] "RemoveContainer" containerID="7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec" Oct 30 19:17:52.262238 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:17:52.258453 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec\": container with ID starting with 7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec not found: ID does not exist" containerID="7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec" Oct 30 19:17:52.262238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.258498 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec"} err="failed to get container status \"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec\": rpc error: code = NotFound desc = could not find container \"7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec\": container with ID starting with 7ebc6d67cec8e7312f0c58daf30f32b349f46d4fa5a826a47cbfe79fee7c3aec not found: ID does not exist" Oct 30 19:17:52.262238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.258526 4789 scope.go:122] "RemoveContainer" containerID="b34857e992f063d8c6073b6ef3f10f96d37ea8db9f8861b9b0533152e8e5a1f2" Oct 30 19:17:52.385840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.385795 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:17:52.389430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.389367 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c4ca9a-openshift-default-96d979775-v7g8z"] Oct 30 19:17:52.639563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.639473 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:17:52.642616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:52.642593 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-9266d"] Oct 30 19:17:53.674652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:53.674622 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a115530-4069-411e-9ab5-289553297b1b" path="/var/lib/kubelet/pods/4a115530-4069-411e-9ab5-289553297b1b/volumes" Oct 30 19:17:53.675341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:53.675321 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" path="/var/lib/kubelet/pods/56ad40a6-a845-4446-b6c6-4d87afa3a746/volumes" Oct 30 19:17:54.926472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.926440 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq"] Oct 30 19:17:54.926979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.926965 4789 cpu_manager.go:414] "containerMap: removing container" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:17:54.926979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.926979 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.926996 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927001 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927012 4789 cpu_manager.go:414] "containerMap: removing container" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927018 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927051 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2add2be2-822e-4882-b263-7937439c9514" containerName="revoke-keys" Oct 30 19:17:54.927070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927056 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2add2be2-822e-4882-b263-7937439c9514" containerName="revoke-keys" Oct 30 19:17:54.927293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927132 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a115530-4069-411e-9ab5-289553297b1b" containerName="istio-proxy" Oct 30 19:17:54.927293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927141 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:17:54.927293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.927154 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2add2be2-822e-4882-b263-7937439c9514" containerName="revoke-keys" Oct 30 19:17:54.930458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.930438 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:54.984081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.984048 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:54.984280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.984132 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:54.984462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.984434 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkqb\" (UniqueName: \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:54.987396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:54.987374 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq"] Oct 30 19:17:55.028533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.028494 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:17:55.028533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.028497 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:17:55.028721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.028544 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:17:55.028721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.028624 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:17:55.086724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.086686 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkqb\" (UniqueName: \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.086882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.086776 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.086882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.086809 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.087565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.087536 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.089394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.089368 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.147242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.147198 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkqb\" (UniqueName: \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\") pod \"maas-api-revoke-keys-e2e-api-a-f44636-99cgq\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.245123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.245090 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:55.626897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:55.626868 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq"] Oct 30 19:17:55.634226 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:17:55.634187 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3818b2b_4806_4472_a065_c15ffad44a25.slice/crio-9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856 WatchSource:0}: Error finding container 9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856: Status 404 returned error can't find the container with id 9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856 Oct 30 19:17:56.257534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:56.257498 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" event={"ID":"a3818b2b-4806-4472-a065-c15ffad44a25","Type":"ContainerStarted","Data":"417744d0e1a7ed504816bdb6ee080743aeedbfed978fcd5294c11c8b25e0f22a"} Oct 30 19:17:56.257911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:56.257561 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" event={"ID":"a3818b2b-4806-4472-a065-c15ffad44a25","Type":"ContainerStarted","Data":"9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856"} Oct 30 19:17:56.366685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:56.366618 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" podStartSLOduration=2.366601354 podStartE2EDuration="2.366601354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:56.366000187 +0000 UTC m=+1912.867910673" watchObservedRunningTime="2026-10-30 19:17:56.366601354 +0000 UTC m=+1912.868511841" Oct 30 19:17:56.888877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:56.888846 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:17:57.267901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:57.267864 4789 generic.go:356] "Generic (PLEG): container finished" podID="a3818b2b-4806-4472-a065-c15ffad44a25" containerID="417744d0e1a7ed504816bdb6ee080743aeedbfed978fcd5294c11c8b25e0f22a" exitCode=0 Oct 30 19:17:57.268343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:57.267950 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" event={"ID":"a3818b2b-4806-4472-a065-c15ffad44a25","Type":"ContainerDied","Data":"417744d0e1a7ed504816bdb6ee080743aeedbfed978fcd5294c11c8b25e0f22a"} Oct 30 19:17:58.440708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.440674 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:58.530287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.530087 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\") pod \"a3818b2b-4806-4472-a065-c15ffad44a25\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " Oct 30 19:17:58.530287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.530186 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\" (UniqueName: \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\") pod \"a3818b2b-4806-4472-a065-c15ffad44a25\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " Oct 30 19:17:58.530911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.530463 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\") pod \"a3818b2b-4806-4472-a065-c15ffad44a25\" (UID: \"a3818b2b-4806-4472-a065-c15ffad44a25\") " Oct 30 19:17:58.536233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.531409 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca" pod "a3818b2b-4806-4472-a065-c15ffad44a25" (UID: "a3818b2b-4806-4472-a065-c15ffad44a25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:58.536233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.534962 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca" pod "a3818b2b-4806-4472-a065-c15ffad44a25" (UID: "a3818b2b-4806-4472-a065-c15ffad44a25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:17:58.555397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.548402 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb" pod "a3818b2b-4806-4472-a065-c15ffad44a25" (UID: "a3818b2b-4806-4472-a065-c15ffad44a25"). InnerVolumeSpecName "kube-api-access-crkqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:58.636358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.635154 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:58.636358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.635182 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkqb\" (UniqueName: \"kubernetes.io/projected/a3818b2b-4806-4472-a065-c15ffad44a25-kube-api-access-crkqb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:58.636358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:58.635194 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3818b2b-4806-4472-a065-c15ffad44a25-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:59.283660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:59.283627 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" Oct 30 19:17:59.283838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:59.283626 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq" event={"ID":"a3818b2b-4806-4472-a065-c15ffad44a25","Type":"ContainerDied","Data":"9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856"} Oct 30 19:17:59.283838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:17:59.283803 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9846f5d8afc7202ca1bbf074127ea201b344c2c90f157a21c1b4967d02be8856" Oct 30 19:18:01.889301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:01.889266 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" podUID="8b54b94e-704c-470b-8f09-0973f183f4c9" containerName="payload-processing" containerID="cri-o://a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80" gracePeriod=25 Oct 30 19:18:02.177153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.177126 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:18:02.282671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.282643 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\") pod \"8b54b94e-704c-470b-8f09-0973f183f4c9\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " Oct 30 19:18:02.282878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.282861 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\" (UniqueName: \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\") pod \"8b54b94e-704c-470b-8f09-0973f183f4c9\" (UID: \"8b54b94e-704c-470b-8f09-0973f183f4c9\") " Oct 30 19:18:02.283090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.283064 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume" pod "8b54b94e-704c-470b-8f09-0973f183f4c9" (UID: "8b54b94e-704c-470b-8f09-0973f183f4c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:02.283529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.283504 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b54b94e-704c-470b-8f09-0973f183f4c9-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:02.285093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.285065 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8" pod "8b54b94e-704c-470b-8f09-0973f183f4c9" (UID: "8b54b94e-704c-470b-8f09-0973f183f4c9"). InnerVolumeSpecName "kube-api-access-hx2g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:02.308916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.308876 4789 generic.go:356] "Generic (PLEG): container finished" podID="8b54b94e-704c-470b-8f09-0973f183f4c9" containerID="a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80" exitCode=0 Oct 30 19:18:02.309157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.309135 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" event={"ID":"8b54b94e-704c-470b-8f09-0973f183f4c9","Type":"ContainerDied","Data":"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80"} Oct 30 19:18:02.309261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.309196 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" event={"ID":"8b54b94e-704c-470b-8f09-0973f183f4c9","Type":"ContainerDied","Data":"78058d3f2af3668a5921349cc5f14d34d4e20466bcc09f7880c51f20ab549267"} Oct 30 19:18:02.309333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.309268 4789 scope.go:122] "RemoveContainer" containerID="a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80" Oct 30 19:18:02.309454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.309437 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq" Oct 30 19:18:02.328582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.328559 4789 scope.go:122] "RemoveContainer" containerID="a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80" Oct 30 19:18:02.328913 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:02.328889 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80\": container with ID starting with a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80 not found: ID does not exist" containerID="a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80" Oct 30 19:18:02.329000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.328928 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80"} err="failed to get container status \"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80\": rpc error: code = NotFound desc = could not find container \"a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80\": container with ID starting with a71c8d742e206f89b84605ded44dc69d8f0af9689b924aff517ed800fe587d80 not found: ID does not exist" Oct 30 19:18:02.385587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.385552 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx2g8\" (UniqueName: \"kubernetes.io/projected/8b54b94e-704c-470b-8f09-0973f183f4c9-kube-api-access-hx2g8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:02.493924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.493885 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:18:02.499116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:02.499086 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0d2ef6-77798dccb6-s24hq"] Oct 30 19:18:03.671844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:03.671811 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b54b94e-704c-470b-8f09-0973f183f4c9" path="/var/lib/kubelet/pods/8b54b94e-704c-470b-8f09-0973f183f4c9/volumes" Oct 30 19:18:05.470249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.470203 4789 scope.go:122] "RemoveContainer" containerID="44d45e4ce3940297391a38799dac7ebfa1334f7de6fbad9ffb6144d2d7b72d7b" Oct 30 19:18:05.481185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.481165 4789 scope.go:122] "RemoveContainer" containerID="7706db4d8c688bced6cbbdc7e331ead961c90e45176f82550abe57450c95b04c" Oct 30 19:18:05.490048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.490031 4789 scope.go:122] "RemoveContainer" containerID="787131e8bfbc92c1462346d3ccb8bd2c469f3751d0cf75b966deb663b5d43873" Oct 30 19:18:05.500987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.500963 4789 scope.go:122] "RemoveContainer" containerID="d718528faf31bd33610e6307d2ff2c26bbf09606d43cd74fbfde90f67a6dd9c1" Oct 30 19:18:05.510880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.510852 4789 scope.go:122] "RemoveContainer" containerID="c895baba160fc3c239fa9ef42302786247d70b64ba33f5aed8ebc118df483870" Oct 30 19:18:05.519029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.519011 4789 scope.go:122] "RemoveContainer" containerID="f882b83731e7b5da02eaa0282ee28e5b9397985cd1f5ee8021b344b75da0a266" Oct 30 19:18:05.527702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.527681 4789 scope.go:122] "RemoveContainer" containerID="3527c255f687537789601019a284f9c6c7e4da4228166fa5555a0a0ea66d079c" Oct 30 19:18:05.536292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:05.536274 4789 scope.go:122] "RemoveContainer" containerID="aaa04bb9f4bc3ebe4c37aea7f8aee54e9afd8527f7d20d4439bc03e5b83de672" Oct 30 19:18:09.108011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:09.107960 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2"] Oct 30 19:18:09.152194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:09.152161 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f1b38f-tmwk2"] Oct 30 19:18:09.668403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:09.668371 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49d9195-becd-4ba9-8e28-43c79dc3e2f7" path="/var/lib/kubelet/pods/a49d9195-becd-4ba9-8e28-43c79dc3e2f7/volumes" Oct 30 19:18:15.872880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:15.872792 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:18:15.873393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:15.873018 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" containerID="cri-o://38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e" gracePeriod=30 Oct 30 19:18:18.196825 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:18.196787 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:18:18.219769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:18.219733 4789 patch_prober.go:29] interesting pod/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:18:18.219923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:18.219794 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:18:21.059263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.059229 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr"] Oct 30 19:18:21.106490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.106448 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e79953b-bnmqr"] Oct 30 19:18:21.142981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.142916 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:18:21.243481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243443 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243480 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243539 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243631 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243768 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243855 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.243902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.243885 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.244086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244061 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.244173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244108 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.244275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244252 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\") pod \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\" (UID: \"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd\") " Oct 30 19:18:21.244357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244107 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.244357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244305 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.244472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244436 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:21.244536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244461 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.244873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244796 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.244873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244823 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.244873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244833 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.244873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244842 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.244873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.244856 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.246123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.246102 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:21.246410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.246390 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:18:21.246683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.246663 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:21.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.246675 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk" pod "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" (UID: "dc89180c-9c3d-44f3-b6ba-b000a3b22bbd"). InnerVolumeSpecName "kube-api-access-lh4sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:21.346171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.346129 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.346171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.346172 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4sk\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-kube-api-access-lh4sk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.346389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.346188 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.346389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.346201 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:21.451536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.451504 4789 generic.go:356] "Generic (PLEG): container finished" podID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerID="38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e" exitCode=0 Oct 30 19:18:21.451708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.451554 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" Oct 30 19:18:21.451708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.451565 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerDied","Data":"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e"} Oct 30 19:18:21.451708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.451615 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj" event={"ID":"dc89180c-9c3d-44f3-b6ba-b000a3b22bbd","Type":"ContainerDied","Data":"fdb4f94329bacd3fcea3d1b4ee1beb853914cb3a397a2331684b7781280ba4f1"} Oct 30 19:18:21.451708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.451638 4789 scope.go:122] "RemoveContainer" containerID="38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e" Oct 30 19:18:21.462602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.462584 4789 scope.go:122] "RemoveContainer" containerID="fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8" Oct 30 19:18:21.471641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.471617 4789 scope.go:122] "RemoveContainer" containerID="38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e" Oct 30 19:18:21.471914 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:21.471893 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e\": container with ID starting with 38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e not found: ID does not exist" containerID="38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e" Oct 30 19:18:21.471968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.471924 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e"} err="failed to get container status \"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e\": rpc error: code = NotFound desc = could not find container \"38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e\": container with ID starting with 38e9b6a82f873c20b0bdf46f8bf8dfb37160f4c10a3a3bad3df8330dbd1d957e not found: ID does not exist" Oct 30 19:18:21.471968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.471943 4789 scope.go:122] "RemoveContainer" containerID="fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8" Oct 30 19:18:21.472201 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:21.472174 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8\": container with ID starting with fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8 not found: ID does not exist" containerID="fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8" Oct 30 19:18:21.472296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.472228 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8"} err="failed to get container status \"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8\": rpc error: code = NotFound desc = could not find container \"fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8\": container with ID starting with fb38e5cd3f0b1812580b29623ed645c4968acb6537a6c13ca689b8b01f69eef8 not found: ID does not exist" Oct 30 19:18:21.606836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.606805 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:18:21.610957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.610933 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f1b38f-openshift-default-7cdfd58ff9-d6bwj"] Oct 30 19:18:21.669101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.669073 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" path="/var/lib/kubelet/pods/dc89180c-9c3d-44f3-b6ba-b000a3b22bbd/volumes" Oct 30 19:18:21.669551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:21.669536 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a5a7e2-5908-475c-bb49-c6862157cbff" path="/var/lib/kubelet/pods/f6a5a7e2-5908-475c-bb49-c6862157cbff/volumes" Oct 30 19:18:32.529169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:32.529126 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:18:32.529645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:32.529381 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" containerID="cri-o://efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed" gracePeriod=30 Oct 30 19:18:38.007650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.007612 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:18:38.099694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099664 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.099872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099734 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.099872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099807 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.099872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099864 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.100035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099905 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.100035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.099976 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.100131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100039 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.100131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100071 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:38.100131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100095 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.100315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100130 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\") pod \"69fbdd87-af90-4c1f-96b5-702155371343\" (UID: \"69fbdd87-af90-4c1f-96b5-702155371343\") " Oct 30 19:18:38.102273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100794 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69fbdd87-af90-4c1f-96b5-702155371343-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.102273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.100793 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:38.102273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.101008 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:38.102273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.101071 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:38.102273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.101122 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:38.102626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.102580 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:18:38.102997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.102972 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:38.103155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.103138 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:18:38.107100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.107074 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt" pod "69fbdd87-af90-4c1f-96b5-702155371343" (UID: "69fbdd87-af90-4c1f-96b5-702155371343"). InnerVolumeSpecName "kube-api-access-dl5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:38.202952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202871 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.202952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202911 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.202952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202927 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl5mt\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-kube-api-access-dl5mt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.202952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202943 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69fbdd87-af90-4c1f-96b5-702155371343-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.203203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202957 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.203203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202971 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.203203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202984 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69fbdd87-af90-4c1f-96b5-702155371343-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.203203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.202997 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69fbdd87-af90-4c1f-96b5-702155371343-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:38.572677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.572591 4789 generic.go:356] "Generic (PLEG): container finished" podID="69fbdd87-af90-4c1f-96b5-702155371343" containerID="efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed" exitCode=0 Oct 30 19:18:38.572677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.572642 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" Oct 30 19:18:38.572866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.572674 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" event={"ID":"69fbdd87-af90-4c1f-96b5-702155371343","Type":"ContainerDied","Data":"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed"} Oct 30 19:18:38.572866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.572761 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6" event={"ID":"69fbdd87-af90-4c1f-96b5-702155371343","Type":"ContainerDied","Data":"704e5649e83737897a19e0affbf856fcb63f385f1f1c37c3d7cf0686f7ec226a"} Oct 30 19:18:38.572866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.572788 4789 scope.go:122] "RemoveContainer" containerID="efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed" Oct 30 19:18:38.583544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.583521 4789 scope.go:122] "RemoveContainer" containerID="efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed" Oct 30 19:18:38.583798 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:38.583778 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed\": container with ID starting with efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed not found: ID does not exist" containerID="efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed" Oct 30 19:18:38.583845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.583807 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed"} err="failed to get container status \"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed\": rpc error: code = NotFound desc = could not find container \"efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed\": container with ID starting with efe014747a3c05150783b3eddb9c0ca2329e680a7ba0b2cb5cc9fdf24cf0e7ed not found: ID does not exist" Oct 30 19:18:38.731587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.731549 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:18:38.777625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:38.777586 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gp7z6"] Oct 30 19:18:39.499508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:39.499479 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb"] Oct 30 19:18:39.504830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:39.504796 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0d2ef6-5fftb"] Oct 30 19:18:39.714178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:39.714144 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2add2be2-822e-4882-b263-7937439c9514" path="/var/lib/kubelet/pods/2add2be2-822e-4882-b263-7937439c9514/volumes" Oct 30 19:18:39.715326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:39.715301 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fbdd87-af90-4c1f-96b5-702155371343" path="/var/lib/kubelet/pods/69fbdd87-af90-4c1f-96b5-702155371343/volumes" Oct 30 19:18:45.165022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.164989 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7"] Oct 30 19:18:45.165642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165624 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8b54b94e-704c-470b-8f09-0973f183f4c9" containerName="payload-processing" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165644 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b54b94e-704c-470b-8f09-0973f183f4c9" containerName="payload-processing" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165657 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165662 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165679 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a3818b2b-4806-4472-a065-c15ffad44a25" containerName="revoke-keys" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165684 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3818b2b-4806-4472-a065-c15ffad44a25" containerName="revoke-keys" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165702 4789 cpu_manager.go:414] "containerMap: removing container" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" Oct 30 19:18:45.165711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165707 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" Oct 30 19:18:45.165940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165791 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3818b2b-4806-4472-a065-c15ffad44a25" containerName="revoke-keys" Oct 30 19:18:45.165940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165801 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ad40a6-a845-4446-b6c6-4d87afa3a746" containerName="istio-proxy" Oct 30 19:18:45.165940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165809 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fbdd87-af90-4c1f-96b5-702155371343" containerName="istio-proxy" Oct 30 19:18:45.165940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165815 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:45.165940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.165823 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b54b94e-704c-470b-8f09-0973f183f4c9" containerName="payload-processing" Oct 30 19:18:45.170661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.170645 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.230765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.230730 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7"] Oct 30 19:18:45.271257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.271203 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:18:45.271426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.271293 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.271491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.271437 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:18:45.271491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.271474 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:18:45.298469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.298441 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.298633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.298476 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.298710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.298693 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxxd\" (UniqueName: \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.400254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.400181 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxxd\" (UniqueName: \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.400394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.400285 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.400394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.400308 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.400920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.400898 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.402733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.402711 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.457665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.457636 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxxd\" (UniqueName: \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\") pod \"maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.484588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.484562 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:45.865458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:45.865376 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7"] Oct 30 19:18:45.872375 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:18:45.872341 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec929a6_9120_4df8_a1a9_34106e6609a4.slice/crio-d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b WatchSource:0}: Error finding container d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b: Status 404 returned error can't find the container with id d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b Oct 30 19:18:46.055908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:46.055870 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6"] Oct 30 19:18:46.059693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:46.059667 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d3acba7-r9qc6"] Oct 30 19:18:46.741828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:46.741791 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" event={"ID":"dec929a6-9120-4df8-a1a9-34106e6609a4","Type":"ContainerStarted","Data":"261b1c118c7060f49d31cfa37a0de8591e6be6f43a7a98eb2a4eb1acb2969f38"} Oct 30 19:18:46.742303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:46.741844 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" event={"ID":"dec929a6-9120-4df8-a1a9-34106e6609a4","Type":"ContainerStarted","Data":"d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b"} Oct 30 19:18:46.859457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:46.859399 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" podStartSLOduration=1.8593838790000001 podStartE2EDuration="1.859383879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:46.858686957 +0000 UTC m=+1963.360597443" watchObservedRunningTime="2026-10-30 19:18:46.859383879 +0000 UTC m=+1963.361294364" Oct 30 19:18:47.668486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:47.668450 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951b817b-f618-488b-9b16-a80c22a0fec2" path="/var/lib/kubelet/pods/951b817b-f618-488b-9b16-a80c22a0fec2/volumes" Oct 30 19:18:47.754048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:47.754007 4789 generic.go:356] "Generic (PLEG): container finished" podID="dec929a6-9120-4df8-a1a9-34106e6609a4" containerID="261b1c118c7060f49d31cfa37a0de8591e6be6f43a7a98eb2a4eb1acb2969f38" exitCode=0 Oct 30 19:18:47.754546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:47.754072 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" event={"ID":"dec929a6-9120-4df8-a1a9-34106e6609a4","Type":"ContainerDied","Data":"261b1c118c7060f49d31cfa37a0de8591e6be6f43a7a98eb2a4eb1acb2969f38"} Oct 30 19:18:48.267128 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:18:48.267080 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:18:48.983231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:48.971830 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:49.069097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.069005 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\") pod \"dec929a6-9120-4df8-a1a9-34106e6609a4\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " Oct 30 19:18:49.069097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.069097 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\") pod \"dec929a6-9120-4df8-a1a9-34106e6609a4\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " Oct 30 19:18:49.069358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.069137 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\" (UniqueName: \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\") pod \"dec929a6-9120-4df8-a1a9-34106e6609a4\" (UID: \"dec929a6-9120-4df8-a1a9-34106e6609a4\") " Oct 30 19:18:49.069541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.069513 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca" pod "dec929a6-9120-4df8-a1a9-34106e6609a4" (UID: "dec929a6-9120-4df8-a1a9-34106e6609a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:18:49.069640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.069622 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:49.071350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.071322 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca" pod "dec929a6-9120-4df8-a1a9-34106e6609a4" (UID: "dec929a6-9120-4df8-a1a9-34106e6609a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:18:49.071514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.071492 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd" pod "dec929a6-9120-4df8-a1a9-34106e6609a4" (UID: "dec929a6-9120-4df8-a1a9-34106e6609a4"). InnerVolumeSpecName "kube-api-access-7gxxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:18:49.204233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.178204 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec929a6-9120-4df8-a1a9-34106e6609a4-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:49.204233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.178265 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxxd\" (UniqueName: \"kubernetes.io/projected/dec929a6-9120-4df8-a1a9-34106e6609a4-kube-api-access-7gxxd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:18:49.771868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.771115 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" Oct 30 19:18:49.771868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.771139 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7" event={"ID":"dec929a6-9120-4df8-a1a9-34106e6609a4","Type":"ContainerDied","Data":"d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b"} Oct 30 19:18:49.771868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:49.771248 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d521227c47438e4a89aaa5800c3e5822db1ea19ffe9526821f21c88b832f390b" Oct 30 19:18:51.125713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.125679 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:18:51.126585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126564 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:51.126585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126585 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:51.126728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126604 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dec929a6-9120-4df8-a1a9-34106e6609a4" containerName="revoke-keys" Oct 30 19:18:51.126728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126610 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec929a6-9120-4df8-a1a9-34106e6609a4" containerName="revoke-keys" Oct 30 19:18:51.126728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126696 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec929a6-9120-4df8-a1a9-34106e6609a4" containerName="revoke-keys" Oct 30 19:18:51.126728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.126708 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc89180c-9c3d-44f3-b6ba-b000a3b22bbd" containerName="istio-proxy" Oct 30 19:18:51.130383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.130365 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.171839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.171810 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq"] Oct 30 19:18:51.176472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.176443 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44636-99cgq"] Oct 30 19:18:51.210909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.210873 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.210917 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.210988 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211045 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211118 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211261 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211289 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211404 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.211474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.211444 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.213122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.213095 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-942c5e85-openshift-default-dockercfg-2q6j6\"" Oct 30 19:18:51.216372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.216351 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:18:51.313186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313140 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313201 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313337 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313378 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313484 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313523 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313559 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313586 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.313826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313786 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.314341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.313991 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.314341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.314024 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.314341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.314290 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.314708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.314685 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.316030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.316010 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.316491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.316474 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.371913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.371879 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.372129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.372105 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.446390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.446345 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:51.669145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.669108 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3818b2b-4806-4472-a065-c15ffad44a25" path="/var/lib/kubelet/pods/a3818b2b-4806-4472-a065-c15ffad44a25/volumes" Oct 30 19:18:51.883037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.882947 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:18:51.889364 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:18:51.889334 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b30a935_2f50_4c3d_a69f_5217e241c766.slice/crio-8184f271aa168da81abbea82ea2e54b8852f39366357c69dd6a0f6c3ea277cc9 WatchSource:0}: Error finding container 8184f271aa168da81abbea82ea2e54b8852f39366357c69dd6a0f6c3ea277cc9: Status 404 returned error can't find the container with id 8184f271aa168da81abbea82ea2e54b8852f39366357c69dd6a0f6c3ea277cc9 Oct 30 19:18:51.892282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.892247 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:18:51.892402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.892327 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:18:51.892402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:51.892363 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:18:52.822310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:52.802416 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" event={"ID":"4b30a935-2f50-4c3d-a69f-5217e241c766","Type":"ContainerStarted","Data":"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62"} Oct 30 19:18:52.822310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:52.802486 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" event={"ID":"4b30a935-2f50-4c3d-a69f-5217e241c766","Type":"ContainerStarted","Data":"8184f271aa168da81abbea82ea2e54b8852f39366357c69dd6a0f6c3ea277cc9"} Oct 30 19:18:52.967269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:52.967199 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podStartSLOduration=1.9671846400000002 podStartE2EDuration="1.96718464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:52.964927674 +0000 UTC m=+1969.466838160" watchObservedRunningTime="2026-10-30 19:18:52.96718464 +0000 UTC m=+1969.469095126" Oct 30 19:18:53.446764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:53.446729 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:53.448003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:53.447971 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:18:53.448120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:53.448019 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:18:54.447406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:54.447363 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:18:54.447936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:54.447423 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:18:55.447177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:55.447143 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:18:55.447413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:55.447203 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:18:56.446682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:56.446653 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:18:56.447148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:56.447125 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:18:57.446870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:57.446835 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:18:57.447372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:57.446891 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 19:18:58.452419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:58.452389 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:58.452868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:58.452853 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:58.453492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:58.453469 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:18:59.597029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:59.596997 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:18:59.597764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:18:59.597706 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" containerID="cri-o://61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1" gracePeriod=30 Oct 30 19:19:00.415158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.415122 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n"] Oct 30 19:19:00.420317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.420289 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.472744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.472714 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:19:00.473267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.473194 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:19:00.476731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.476692 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n"] Oct 30 19:19:00.497323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.494776 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:19:00.505262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.502448 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:19:00.520718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.520449 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcc2\" (UniqueName: \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.520718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.520589 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.520995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.520733 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.621746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.621711 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcc2\" (UniqueName: \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.622150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.621823 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.622150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.621874 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.622564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.622534 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.624419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.624393 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.696272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.695585 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcc2\" (UniqueName: \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\") pod \"maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:00.739662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:00.739621 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:01.125258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:01.125152 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n"] Oct 30 19:19:01.135535 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:19:01.135477 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d1de61_8760_4576_9a48_85dbb7b23302.slice/crio-41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2 WatchSource:0}: Error finding container 41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2: Status 404 returned error can't find the container with id 41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2 Oct 30 19:19:01.920173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:01.920138 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" event={"ID":"d7d1de61-8760-4576-9a48-85dbb7b23302","Type":"ContainerStarted","Data":"1e9fdac30ecd61b9e6f4f435b639c64ddac3dd1a53787a6ad7b0279c6d73f33c"} Oct 30 19:19:01.920598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:01.920197 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" event={"ID":"d7d1de61-8760-4576-9a48-85dbb7b23302","Type":"ContainerStarted","Data":"41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2"} Oct 30 19:19:02.032175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:02.032128 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" podStartSLOduration=2.032114299 podStartE2EDuration="2.032114299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:02.030016104 +0000 UTC m=+1978.531926589" watchObservedRunningTime="2026-10-30 19:19:02.032114299 +0000 UTC m=+1978.534024785" Oct 30 19:19:02.929088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:02.929049 4789 generic.go:356] "Generic (PLEG): container finished" podID="d7d1de61-8760-4576-9a48-85dbb7b23302" containerID="1e9fdac30ecd61b9e6f4f435b639c64ddac3dd1a53787a6ad7b0279c6d73f33c" exitCode=0 Oct 30 19:19:02.929587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:02.929129 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" event={"ID":"d7d1de61-8760-4576-9a48-85dbb7b23302","Type":"ContainerDied","Data":"1e9fdac30ecd61b9e6f4f435b639c64ddac3dd1a53787a6ad7b0279c6d73f33c"} Oct 30 19:19:03.200612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:03.200578 4789 patch_prober.go:29] interesting pod/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:19:03.200775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:03.200634 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:19:04.102561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.102537 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:04.186396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.186371 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\" (UniqueName: \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\") pod \"d7d1de61-8760-4576-9a48-85dbb7b23302\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " Oct 30 19:19:04.186522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.186487 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\") pod \"d7d1de61-8760-4576-9a48-85dbb7b23302\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " Oct 30 19:19:04.186566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.186530 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\") pod \"d7d1de61-8760-4576-9a48-85dbb7b23302\" (UID: \"d7d1de61-8760-4576-9a48-85dbb7b23302\") " Oct 30 19:19:04.186862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.186835 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca" pod "d7d1de61-8760-4576-9a48-85dbb7b23302" (UID: "d7d1de61-8760-4576-9a48-85dbb7b23302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:04.187116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.187093 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:04.188499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.188451 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2" pod "d7d1de61-8760-4576-9a48-85dbb7b23302" (UID: "d7d1de61-8760-4576-9a48-85dbb7b23302"). InnerVolumeSpecName "kube-api-access-mwcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:04.188556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.188530 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca" pod "d7d1de61-8760-4576-9a48-85dbb7b23302" (UID: "d7d1de61-8760-4576-9a48-85dbb7b23302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:04.288521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.288489 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwcc2\" (UniqueName: \"kubernetes.io/projected/d7d1de61-8760-4576-9a48-85dbb7b23302-kube-api-access-mwcc2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:04.288521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.288517 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d1de61-8760-4576-9a48-85dbb7b23302-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:04.927594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.927566 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:19:04.950367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.950337 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" Oct 30 19:19:04.950496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.950386 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n" event={"ID":"d7d1de61-8760-4576-9a48-85dbb7b23302","Type":"ContainerDied","Data":"41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2"} Oct 30 19:19:04.950496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.950441 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41ff47a04884037ba8c9437d436f56036e49928b82f235c486dc7d325dc19cb2" Oct 30 19:19:04.952162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.952134 4789 generic.go:356] "Generic (PLEG): container finished" podID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerID="61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1" exitCode=0 Oct 30 19:19:04.952288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.952176 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerDied","Data":"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1"} Oct 30 19:19:04.952288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.952189 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" Oct 30 19:19:04.952288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.952224 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w" event={"ID":"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05","Type":"ContainerDied","Data":"9849c260f487f2c80baddcafbe077013edae6b4142f9c5db3d2fadd8bdc471c8"} Oct 30 19:19:04.952288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.952259 4789 scope.go:122] "RemoveContainer" containerID="61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1" Oct 30 19:19:04.965122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.965088 4789 scope.go:122] "RemoveContainer" containerID="d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937" Oct 30 19:19:04.974779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.974755 4789 scope.go:122] "RemoveContainer" containerID="61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1" Oct 30 19:19:04.975062 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:04.975042 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1\": container with ID starting with 61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1 not found: ID does not exist" containerID="61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1" Oct 30 19:19:04.975126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.975070 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1"} err="failed to get container status \"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1\": rpc error: code = NotFound desc = could not find container \"61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1\": container with ID starting with 61f52cdd686d3ef62e90a8693f8ac29e2fe6cd67f558a9b42887ff121366bfa1 not found: ID does not exist" Oct 30 19:19:04.975126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.975086 4789 scope.go:122] "RemoveContainer" containerID="d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937" Oct 30 19:19:04.975412 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:04.975394 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937\": container with ID starting with d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937 not found: ID does not exist" containerID="d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937" Oct 30 19:19:04.975480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:04.975417 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937"} err="failed to get container status \"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937\": rpc error: code = NotFound desc = could not find container \"d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937\": container with ID starting with d97ce0001fceaca84fb13b9565a7f592e75fd2111ae3c2d73d1c0f285d3af937 not found: ID does not exist" Oct 30 19:19:05.005364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.005336 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.006004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.005984 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.006471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.006455 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.007468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.005846 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:05.007817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.006385 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:05.007917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.007767 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:05.008011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.007428 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.008351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.008332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.008830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.008812 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.009290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.009272 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.010841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.009886 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:05.054448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.015150 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:05.054448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.015413 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:05.054448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.017750 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "kube-api-access-grhhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:05.054448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.026414 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:05.054762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.054733 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.062350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.062321 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\") pod \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\" (UID: \"5836ee2e-5b1b-4c2c-a751-c5efaa41cc05\") " Oct 30 19:19:05.068761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.063138 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.068959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.068939 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069074 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069189 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069305 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grhhv\" (UniqueName: \"kubernetes.io/projected/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-kube-api-access-grhhv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069423 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069524 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.069620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.069608 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.124282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.101401 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo" pod "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" (UID: "5836ee2e-5b1b-4c2c-a751-c5efaa41cc05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:19:05.171309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.171234 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:05.367431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.367394 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:19:05.371799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.371764 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f44636-openshift-default-698f4fc695-mlr9w"] Oct 30 19:19:05.615914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.615883 4789 scope.go:122] "RemoveContainer" containerID="4f8bbb232dc38d5df4bb745fa99650b47cb46dd89f4d93df9132ccee5a9beb2d" Oct 30 19:19:05.668632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:05.668609 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" path="/var/lib/kubelet/pods/5836ee2e-5b1b-4c2c-a751-c5efaa41cc05/volumes" Oct 30 19:19:06.269916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.269885 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:19:06.270563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270544 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.270629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270564 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.270629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270585 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.270629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270591 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.270629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270623 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d7d1de61-8760-4576-9a48-85dbb7b23302" containerName="revoke-keys" Oct 30 19:19:06.270629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270628 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d1de61-8760-4576-9a48-85dbb7b23302" containerName="revoke-keys" Oct 30 19:19:06.270792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270707 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d1de61-8760-4576-9a48-85dbb7b23302" containerName="revoke-keys" Oct 30 19:19:06.270792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270715 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.270792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.270724 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5836ee2e-5b1b-4c2c-a751-c5efaa41cc05" containerName="istio-proxy" Oct 30 19:19:06.275878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.275859 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.360134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.359836 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:19:06.384715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.384690 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.384881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.384725 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.384881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.384746 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.384881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.384818 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.385063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.384916 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.385063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.385014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.385063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.385042 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.385253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.385198 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.385313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.385287 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjl86\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487201 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487303 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487334 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487382 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487446 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487504 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487537 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487887 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.487957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.487939 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjl86\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.488652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.488626 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.488799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.488772 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.488921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.488895 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.489106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.489080 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.489263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.489245 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.490977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.490932 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.491558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.491533 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.545785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.545716 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.545893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.545811 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjl86\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:06.589190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:06.589167 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:07.026944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.026914 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:19:07.033688 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:19:07.033659 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5d8315_3907_4bc3_9d24_edcb3eb74658.slice/crio-b4a7ea9c12c020a7768642c1a931ffc2a06c4bfc1013043476377c24a892aff9 WatchSource:0}: Error finding container b4a7ea9c12c020a7768642c1a931ffc2a06c4bfc1013043476377c24a892aff9: Status 404 returned error can't find the container with id b4a7ea9c12c020a7768642c1a931ffc2a06c4bfc1013043476377c24a892aff9 Oct 30 19:19:07.036753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.036714 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:07.036853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.036805 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:07.036904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.036851 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:07.974560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.974516 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" event={"ID":"4e5d8315-3907-4bc3-9d24-edcb3eb74658","Type":"ContainerStarted","Data":"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c"} Oct 30 19:19:07.974938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:07.974618 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" event={"ID":"4e5d8315-3907-4bc3-9d24-edcb3eb74658","Type":"ContainerStarted","Data":"b4a7ea9c12c020a7768642c1a931ffc2a06c4bfc1013043476377c24a892aff9"} Oct 30 19:19:08.147203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.147127 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podStartSLOduration=2.14710802 podStartE2EDuration="2.14710802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:08.134374103 +0000 UTC m=+1984.636284592" watchObservedRunningTime="2026-10-30 19:19:08.14710802 +0000 UTC m=+1984.649018507" Oct 30 19:19:08.226963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.226868 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7"] Oct 30 19:19:08.230107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.230078 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-942c5e85-6n4r7"] Oct 30 19:19:08.590323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.590243 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:08.596241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.595312 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:19:08.596241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:08.595380 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:19:09.590383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:09.590352 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:19:09.590795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:09.590404 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:19:09.668597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:09.668563 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec929a6-9120-4df8-a1a9-34106e6609a4" path="/var/lib/kubelet/pods/dec929a6-9120-4df8-a1a9-34106e6609a4/volumes" Oct 30 19:19:10.589925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:10.589891 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:19:10.590083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:10.589946 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:19:11.590487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:11.590455 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:19:11.590860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:11.590506 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:19:11.942196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:11.942165 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:19:12.589724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:12.589694 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:19:12.589876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:12.589744 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:19:13.645176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:13.645123 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:19:13.727544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:13.727515 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:13.727838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:13.727817 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:13.729169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:13.729149 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:19:16.943059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:16.943017 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerName="payload-pre-processing" containerID="cri-o://202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6" gracePeriod=25 Oct 30 19:19:17.218714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.218686 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:19:17.337422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.337379 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\" (UniqueName: \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\") pod \"b8b87e22-9b1f-4d89-9910-0a29b886414c\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " Oct 30 19:19:17.338102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.338076 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\") pod \"b8b87e22-9b1f-4d89-9910-0a29b886414c\" (UID: \"b8b87e22-9b1f-4d89-9910-0a29b886414c\") " Oct 30 19:19:17.338539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.338513 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume" pod "b8b87e22-9b1f-4d89-9910-0a29b886414c" (UID: "b8b87e22-9b1f-4d89-9910-0a29b886414c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:17.338781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.338728 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b87e22-9b1f-4d89-9910-0a29b886414c-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:17.344846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.344820 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l" pod "b8b87e22-9b1f-4d89-9910-0a29b886414c" (UID: "b8b87e22-9b1f-4d89-9910-0a29b886414c"). InnerVolumeSpecName "kube-api-access-hvh9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:17.439854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:17.439828 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvh9l\" (UniqueName: \"kubernetes.io/projected/b8b87e22-9b1f-4d89-9910-0a29b886414c-kube-api-access-hvh9l\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:18.063476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.063445 4789 generic.go:356] "Generic (PLEG): container finished" podID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerID="202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6" exitCode=0 Oct 30 19:19:18.063982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.063490 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" Oct 30 19:19:18.063982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.063510 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" event={"ID":"b8b87e22-9b1f-4d89-9910-0a29b886414c","Type":"ContainerDied","Data":"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6"} Oct 30 19:19:18.063982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.063565 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" event={"ID":"b8b87e22-9b1f-4d89-9910-0a29b886414c","Type":"ContainerDied","Data":"e49ca3c8aec79213868d02b1e99613156cce06ee3073a5fd193a044017179d2d"} Oct 30 19:19:18.063982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.063588 4789 scope.go:122] "RemoveContainer" containerID="202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6" Oct 30 19:19:18.073399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.073384 4789 scope.go:122] "RemoveContainer" containerID="202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6" Oct 30 19:19:18.073637 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:18.073618 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6\": container with ID starting with 202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6 not found: ID does not exist" containerID="202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6" Oct 30 19:19:18.073691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.073644 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6"} err="failed to get container status \"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6\": rpc error: code = NotFound desc = could not find container \"202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6\": container with ID starting with 202f1226135660d74dbca1ffd1699b6afd7ecd2373613d5bef10717848c563d6 not found: ID does not exist" Oct 30 19:19:18.161193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.161162 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.130:50052\" within 1s: context deadline exceeded" Oct 30 19:19:18.161331 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:19:18.161245 4789 logging.go:55] [core] [Channel #676 SubChannel #677]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.130:50052", ServerName: "10.132.0.130:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.130:50052: operation was canceled" Oct 30 19:19:18.196012 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:18.195987 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:19:18.218692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.218667 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:19:18.264784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:18.264762 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ec7ea-6c8dd4db55-85dgs"] Oct 30 19:19:19.668224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:19.668185 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" path="/var/lib/kubelet/pods/b8b87e22-9b1f-4d89-9910-0a29b886414c/volumes" Oct 30 19:19:21.393798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.393763 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:19:21.394426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.394408 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerName="payload-pre-processing" Oct 30 19:19:21.394481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.394427 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerName="payload-pre-processing" Oct 30 19:19:21.394556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.394546 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b87e22-9b1f-4d89-9910-0a29b886414c" containerName="payload-pre-processing" Oct 30 19:19:21.399189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.399168 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.484710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484676 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.484843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484733 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mpr\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.484843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484812 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.484917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484864 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.484917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484895 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.485007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.484993 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.485042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.485022 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.485115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.485095 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.485153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.485124 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.499051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.499026 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:19:21.586594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586561 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.586757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586627 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mpr\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.586757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586649 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.586757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586686 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.586921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586829 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.586921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586905 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587019 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.586961 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587019 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587083 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587108 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587262 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587481 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587541 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.587639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.587585 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.588973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.588952 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.589370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.589352 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.644443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.644373 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.644569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.644539 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mpr\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\") pod \"e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:21.725477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:21.725449 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:22.168811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:22.168777 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:19:22.182699 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:19:22.182662 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1364b9c9_f8ac_450d_857f_599296acf475.slice/crio-351b130f150d2f4b1318e93c6e2d5cf0991cfb3fbbd7d073a00c982e45b7af80 WatchSource:0}: Error finding container 351b130f150d2f4b1318e93c6e2d5cf0991cfb3fbbd7d073a00c982e45b7af80: Status 404 returned error can't find the container with id 351b130f150d2f4b1318e93c6e2d5cf0991cfb3fbbd7d073a00c982e45b7af80 Oct 30 19:19:22.186354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:22.186318 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:22.186465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:22.186410 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:22.186536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:22.186484 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:19:23.111149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.107624 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" event={"ID":"1364b9c9-f8ac-450d-857f-599296acf475","Type":"ContainerStarted","Data":"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec"} Oct 30 19:19:23.111149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.107725 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" event={"ID":"1364b9c9-f8ac-450d-857f-599296acf475","Type":"ContainerStarted","Data":"351b130f150d2f4b1318e93c6e2d5cf0991cfb3fbbd7d073a00c982e45b7af80"} Oct 30 19:19:23.279021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.278966 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podStartSLOduration=2.278951892 podStartE2EDuration="2.278951892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:23.27515552 +0000 UTC m=+1999.777066008" watchObservedRunningTime="2026-10-30 19:19:23.278951892 +0000 UTC m=+1999.780862378" Oct 30 19:19:23.728234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.726742 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:23.728446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.728366 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:19:23.728446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:23.728424 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 19:19:24.726811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:24.726778 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:19:24.727219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:24.726837 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 19:19:25.726483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:25.726453 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:19:25.726675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:25.726505 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 19:19:26.726343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:26.726302 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:19:26.726727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:26.726366 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 19:19:27.726657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:27.726622 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:19:27.727123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:27.726681 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 19:19:28.677082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:28.677036 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" podUID="7e4a6351-6a80-46da-a49e-5d72566ee749" containerName="main" containerID="cri-o://f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280" gracePeriod=45 Oct 30 19:19:28.735361 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:28.735315 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4a6351_6a80_46da_a49e_5d72566ee749.slice/crio-conmon-f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:19:28.735884 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:28.735375 4789 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4a6351_6a80_46da_a49e_5d72566ee749.slice/crio-conmon-f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:19:28.922713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:28.922636 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:28.923822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:28.923339 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:28.927003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:28.926979 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:19:28.957040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:28.957019 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.031557 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.032142 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.032242 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\" (UniqueName: \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.032330 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.032422 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.032488 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\") pod \"7e4a6351-6a80-46da-a49e-5d72566ee749\" (UID: \"7e4a6351-6a80-46da-a49e-5d72566ee749\") " Oct 30 19:19:29.034717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.034666 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:29.035791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.034911 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:29.037800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.036156 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:29.037800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.036350 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:29.040736 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.039379 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:29.043484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.043069 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7" pod "7e4a6351-6a80-46da-a49e-5d72566ee749" (UID: "7e4a6351-6a80-46da-a49e-5d72566ee749"). InnerVolumeSpecName "kube-api-access-gr4g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:29.135333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135305 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4a6351-6a80-46da-a49e-5d72566ee749-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.135333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135330 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4g7\" (UniqueName: \"kubernetes.io/projected/7e4a6351-6a80-46da-a49e-5d72566ee749-kube-api-access-gr4g7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.135333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135341 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.135559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135349 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.135559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135358 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.135559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.135366 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4a6351-6a80-46da-a49e-5d72566ee749-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:29.176169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.176140 4789 generic.go:356] "Generic (PLEG): container finished" podID="7e4a6351-6a80-46da-a49e-5d72566ee749" containerID="f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280" exitCode=0 Oct 30 19:19:29.176329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.176189 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" Oct 30 19:19:29.176329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.176247 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" event={"ID":"7e4a6351-6a80-46da-a49e-5d72566ee749","Type":"ContainerDied","Data":"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280"} Oct 30 19:19:29.176329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.176282 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6" event={"ID":"7e4a6351-6a80-46da-a49e-5d72566ee749","Type":"ContainerDied","Data":"73551bb35ae53a3e20fd2f3b075ac0059438e8cb9a460bee64a0d8c25ed7a021"} Oct 30 19:19:29.176329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.176298 4789 scope.go:122] "RemoveContainer" containerID="f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280" Oct 30 19:19:29.186845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.186829 4789 scope.go:122] "RemoveContainer" containerID="f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280" Oct 30 19:19:29.187108 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:29.187088 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280\": container with ID starting with f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280 not found: ID does not exist" containerID="f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280" Oct 30 19:19:29.187157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.187117 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280"} err="failed to get container status \"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280\": rpc error: code = NotFound desc = could not find container \"f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280\": container with ID starting with f2b178096e1e4d3c84acfea1c6fb6504754c270df4483904bdf207be4798c280 not found: ID does not exist" Oct 30 19:19:29.428943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.428901 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:19:29.432048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.432019 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-942c5e85/e2e-lifecycle-model-942c5e85-kserve-5d7b5b877-w2ps6"] Oct 30 19:19:29.669134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:29.669060 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4a6351-6a80-46da-a49e-5d72566ee749" path="/var/lib/kubelet/pods/7e4a6351-6a80-46da-a49e-5d72566ee749/volumes" Oct 30 19:19:48.196659 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:19:48.196569 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:19:58.423072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.423029 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:19:58.423677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.423287 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" containerID="cri-o://f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c" gracePeriod=30 Oct 30 19:19:58.469112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.469086 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:19:58.469317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.469295 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" containerID="cri-o://b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec" gracePeriod=30 Oct 30 19:19:58.472016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.471980 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:19:58.472242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.472188 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" containerID="cri-o://3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62" gracePeriod=30 Oct 30 19:19:58.730979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.730942 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:19:58.731160 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.731011 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:19:58.925329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.925288 4789 patch_prober.go:29] interesting pod/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:19:58.925518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:19:58.925355 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:20:03.809226 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.809186 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:20:03.824378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.824353 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:20:03.853884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.853862 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:20:03.989515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989477 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.989691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989526 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.989691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989579 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.989691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989600 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.989691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989632 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.989691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989660 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989698 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989727 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989774 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989804 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989786 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989854 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989872 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989924 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.989971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989961 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.989990 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990038 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990125 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990143 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990156 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990190 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990251 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990432 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.990487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990291 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990496 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990536 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990569 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990594 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\") pod \"1364b9c9-f8ac-450d-857f-599296acf475\" (UID: \"1364b9c9-f8ac-450d-857f-599296acf475\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990683 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990755 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\") pod \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\" (UID: \"4e5d8315-3907-4bc3-9d24-edcb3eb74658\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990777 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990798 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\") pod \"4b30a935-2f50-4c3d-a69f-5217e241c766\" (UID: \"4b30a935-2f50-4c3d-a69f-5217e241c766\") " Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.990975 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.991520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.991311 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:03.992850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.992828 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:03.993379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.993105 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.993379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.993267 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.993525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.993436 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.993586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.993521 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:03.994288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.994261 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.994775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.994753 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.994895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.994843 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:03.995138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.995075 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.997859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.997825 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "kube-api-access-v7mpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:03.998151 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.998120 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:03.998648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.998620 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:03.998738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.998655 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:03.999322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:03.999302 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86" pod "4e5d8315-3907-4bc3-9d24-edcb3eb74658" (UID: "4e5d8315-3907-4bc3-9d24-edcb3eb74658"). InnerVolumeSpecName "kube-api-access-cjl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:04.000321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000294 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:04.000492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000464 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000496 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000513 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000528 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000542 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000555 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1364b9c9-f8ac-450d-857f-599296acf475-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000568 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000582 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000596 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.000610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000610 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000624 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000639 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000653 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mpr\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-kube-api-access-v7mpr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000667 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1364b9c9-f8ac-450d-857f-599296acf475-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000681 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjl86\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-kube-api-access-cjl86\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000694 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000707 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000721 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000735 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000748 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5d8315-3907-4bc3-9d24-edcb3eb74658-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000761 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b30a935-2f50-4c3d-a69f-5217e241c766-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.000774 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5d8315-3907-4bc3-9d24-edcb3eb74658-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.001957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.001929 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:04.006539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.006512 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:04.006630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.006520 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:04.006630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.006523 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz" pod "4b30a935-2f50-4c3d-a69f-5217e241c766" (UID: "4b30a935-2f50-4c3d-a69f-5217e241c766"). InnerVolumeSpecName "kube-api-access-h2sxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:04.007615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.007595 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo" pod "1364b9c9-f8ac-450d-857f-599296acf475" (UID: "1364b9c9-f8ac-450d-857f-599296acf475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:04.104252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.104163 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.104252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.104193 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2sxz\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-kube-api-access-h2sxz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.104252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.104221 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1364b9c9-f8ac-450d-857f-599296acf475-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.104252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.104234 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1364b9c9-f8ac-450d-857f-599296acf475-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.104252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.104247 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b30a935-2f50-4c3d-a69f-5217e241c766-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:04.422275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.422171 4789 generic.go:356] "Generic (PLEG): container finished" podID="1364b9c9-f8ac-450d-857f-599296acf475" containerID="b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec" exitCode=0 Oct 30 19:20:04.422275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.422241 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" Oct 30 19:20:04.422275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.422248 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" event={"ID":"1364b9c9-f8ac-450d-857f-599296acf475","Type":"ContainerDied","Data":"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec"} Oct 30 19:20:04.422707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.422335 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2" event={"ID":"1364b9c9-f8ac-450d-857f-599296acf475","Type":"ContainerDied","Data":"351b130f150d2f4b1318e93c6e2d5cf0991cfb3fbbd7d073a00c982e45b7af80"} Oct 30 19:20:04.422707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.422361 4789 scope.go:122] "RemoveContainer" containerID="b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec" Oct 30 19:20:04.424180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.424156 4789 generic.go:356] "Generic (PLEG): container finished" podID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerID="f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c" exitCode=0 Oct 30 19:20:04.424335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.424248 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" Oct 30 19:20:04.424335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.424257 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" event={"ID":"4e5d8315-3907-4bc3-9d24-edcb3eb74658","Type":"ContainerDied","Data":"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c"} Oct 30 19:20:04.424335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.424311 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr" event={"ID":"4e5d8315-3907-4bc3-9d24-edcb3eb74658","Type":"ContainerDied","Data":"b4a7ea9c12c020a7768642c1a931ffc2a06c4bfc1013043476377c24a892aff9"} Oct 30 19:20:04.428675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.428653 4789 generic.go:356] "Generic (PLEG): container finished" podID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerID="3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62" exitCode=0 Oct 30 19:20:04.428794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.428703 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" Oct 30 19:20:04.428794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.428707 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" event={"ID":"4b30a935-2f50-4c3d-a69f-5217e241c766","Type":"ContainerDied","Data":"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62"} Oct 30 19:20:04.428794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.428749 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw" event={"ID":"4b30a935-2f50-4c3d-a69f-5217e241c766","Type":"ContainerDied","Data":"8184f271aa168da81abbea82ea2e54b8852f39366357c69dd6a0f6c3ea277cc9"} Oct 30 19:20:04.435110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.435066 4789 scope.go:122] "RemoveContainer" containerID="b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec" Oct 30 19:20:04.435430 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:04.435406 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec\": container with ID starting with b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec not found: ID does not exist" containerID="b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec" Oct 30 19:20:04.435599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.435570 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec"} err="failed to get container status \"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec\": rpc error: code = NotFound desc = could not find container \"b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec\": container with ID starting with b7d171f0116bb8540251d412d9082822704dece4e4a61340f4ce9d8a1f0b46ec not found: ID does not exist" Oct 30 19:20:04.435736 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.435721 4789 scope.go:122] "RemoveContainer" containerID="f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c" Oct 30 19:20:04.446526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.446503 4789 scope.go:122] "RemoveContainer" containerID="f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c" Oct 30 19:20:04.446817 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:04.446795 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c\": container with ID starting with f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c not found: ID does not exist" containerID="f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c" Oct 30 19:20:04.446902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.446821 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c"} err="failed to get container status \"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c\": rpc error: code = NotFound desc = could not find container \"f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c\": container with ID starting with f53d2369da0f837fe14f5d478a5318ab1b145c7538615096fe988ef7040f432c not found: ID does not exist" Oct 30 19:20:04.446902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.446838 4789 scope.go:122] "RemoveContainer" containerID="3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62" Oct 30 19:20:04.475739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.475722 4789 scope.go:122] "RemoveContainer" containerID="3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62" Oct 30 19:20:04.476007 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:04.475982 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62\": container with ID starting with 3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62 not found: ID does not exist" containerID="3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62" Oct 30 19:20:04.476077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.476017 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62"} err="failed to get container status \"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62\": rpc error: code = NotFound desc = could not find container \"3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62\": container with ID starting with 3c5b98288e4a05671e78dab5490851695a89ebd858b900f3619928dd1f812b62 not found: ID does not exist" Oct 30 19:20:04.578317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.578292 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:20:04.584351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.584329 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-xtcqw"] Oct 30 19:20:04.689360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.689337 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:20:04.694666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.694638 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-8rdd2"] Oct 30 19:20:04.796330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.796297 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:20:04.803468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:04.803437 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-942c5e85-openshift-default-845cb4548f-cp8fr"] Oct 30 19:20:05.660192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:05.660160 4789 scope.go:122] "RemoveContainer" containerID="d5d00260037bb7f398d805cf8849841d4537746c9674d69b322799332164e9ae" Oct 30 19:20:05.667463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:05.667429 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1364b9c9-f8ac-450d-857f-599296acf475" path="/var/lib/kubelet/pods/1364b9c9-f8ac-450d-857f-599296acf475/volumes" Oct 30 19:20:05.667934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:05.667917 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" path="/var/lib/kubelet/pods/4b30a935-2f50-4c3d-a69f-5217e241c766/volumes" Oct 30 19:20:05.668378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:05.668363 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" path="/var/lib/kubelet/pods/4e5d8315-3907-4bc3-9d24-edcb3eb74658/volumes" Oct 30 19:20:05.669717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:05.669702 4789 scope.go:122] "RemoveContainer" containerID="05209479dc6cd1d9b282f3cc23c0287795391f469fce9d11a03d5c7758197a43" Oct 30 19:20:07.381037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381006 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:20:07.381531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381516 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" Oct 30 19:20:07.381587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381532 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" Oct 30 19:20:07.381587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381542 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" Oct 30 19:20:07.381587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381548 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" Oct 30 19:20:07.381587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381580 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" Oct 30 19:20:07.381587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381585 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381593 4789 cpu_manager.go:414] "containerMap: removing container" podUID="7e4a6351-6a80-46da-a49e-5d72566ee749" containerName="main" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381598 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4a6351-6a80-46da-a49e-5d72566ee749" containerName="main" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381676 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1364b9c9-f8ac-450d-857f-599296acf475" containerName="istio-proxy" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381682 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4a6351-6a80-46da-a49e-5d72566ee749" containerName="main" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381694 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5d8315-3907-4bc3-9d24-edcb3eb74658" containerName="istio-proxy" Oct 30 19:20:07.381744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.381701 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b30a935-2f50-4c3d-a69f-5217e241c766" containerName="istio-proxy" Oct 30 19:20:07.386953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.386934 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.436596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.436573 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-324ebe\"" Oct 30 19:20:07.450238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.449948 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvnw\" (UniqueName: \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.450238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.450118 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.485764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.485735 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:20:07.564011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.552916 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvnw\" (UniqueName: \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.564492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.564460 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.565599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.565561 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.646953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.646497 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvnw\" (UniqueName: \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\") pod \"payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:07.702752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:07.702718 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:08.088775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:08.088745 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:20:08.094932 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:20:08.094900 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8105bd64_c702_4a7e_959b_2de393c39690.slice/crio-b2be9aebb2fc750709bc4d2130f493e61c5a7d39873e7129ae910f8607f77bb0 WatchSource:0}: Error finding container b2be9aebb2fc750709bc4d2130f493e61c5a7d39873e7129ae910f8607f77bb0: Status 404 returned error can't find the container with id b2be9aebb2fc750709bc4d2130f493e61c5a7d39873e7129ae910f8607f77bb0 Oct 30 19:20:08.463916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:08.463878 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" event={"ID":"8105bd64-c702-4a7e-959b-2de393c39690","Type":"ContainerStarted","Data":"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4"} Oct 30 19:20:08.464336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:08.463941 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" event={"ID":"8105bd64-c702-4a7e-959b-2de393c39690","Type":"ContainerStarted","Data":"b2be9aebb2fc750709bc4d2130f493e61c5a7d39873e7129ae910f8607f77bb0"} Oct 30 19:20:08.464336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:08.463974 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:08.577007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:08.576954 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" podStartSLOduration=1.576940307 podStartE2EDuration="1.576940307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:08.573156416 +0000 UTC m=+2045.075066901" watchObservedRunningTime="2026-10-30 19:20:08.576940307 +0000 UTC m=+2045.078850829" Oct 30 19:20:18.196613 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:18.196578 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:20:19.473595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:19.473569 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:20:46.293024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:46.292991 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n"] Oct 30 19:20:46.338672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:46.338642 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ec7ea-8b24n"] Oct 30 19:20:47.668308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:47.668277 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d1de61-8760-4576-9a48-85dbb7b23302" path="/var/lib/kubelet/pods/d7d1de61-8760-4576-9a48-85dbb7b23302/volumes" Oct 30 19:20:48.196409 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:48.196376 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:20:53.777999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:53.777954 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:20:53.778679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:53.778283 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" containerID="cri-o://91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0" gracePeriod=30 Oct 30 19:20:59.089526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.089502 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:20:59.123175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123148 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123182 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123240 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123274 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123305 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123376 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123401 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123539 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\") pod \"3db188f4-4939-4295-acdf-ee5077b4c1f4\" (UID: \"3db188f4-4939-4295-acdf-ee5077b4c1f4\") " Oct 30 19:20:59.123805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.123773 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:59.124406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124031 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:59.124406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124101 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:59.124406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124363 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:59.124406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124375 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:59.124675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124561 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.124675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124582 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.124675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124595 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.124675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124609 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3db188f4-4939-4295-acdf-ee5077b4c1f4-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.124675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.124623 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.125771 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.125734 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:59.126194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.126163 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:59.127168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.127140 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:59.127877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.127849 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c" pod "3db188f4-4939-4295-acdf-ee5077b4c1f4" (UID: "3db188f4-4939-4295-acdf-ee5077b4c1f4"). InnerVolumeSpecName "kube-api-access-jgc8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:59.225748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.225721 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.225855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.225752 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.225855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.225767 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.225855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.225781 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgc8c\" (UniqueName: \"kubernetes.io/projected/3db188f4-4939-4295-acdf-ee5077b4c1f4-kube-api-access-jgc8c\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:59.854280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.854229 4789 generic.go:356] "Generic (PLEG): container finished" podID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerID="91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0" exitCode=0 Oct 30 19:20:59.854444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.854290 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" Oct 30 19:20:59.854444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.854300 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerDied","Data":"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0"} Oct 30 19:20:59.854444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.854357 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc" event={"ID":"3db188f4-4939-4295-acdf-ee5077b4c1f4","Type":"ContainerDied","Data":"a608675810a74fb81fe634d98e091032fa67819dc75457b6f658fb754deeeee1"} Oct 30 19:20:59.854444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.854384 4789 scope.go:122] "RemoveContainer" containerID="91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0" Oct 30 19:20:59.864171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.864153 4789 scope.go:122] "RemoveContainer" containerID="972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68" Oct 30 19:20:59.873037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.873022 4789 scope.go:122] "RemoveContainer" containerID="91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0" Oct 30 19:20:59.873312 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:59.873288 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0\": container with ID starting with 91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0 not found: ID does not exist" containerID="91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0" Oct 30 19:20:59.873408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.873317 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0"} err="failed to get container status \"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0\": rpc error: code = NotFound desc = could not find container \"91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0\": container with ID starting with 91c8a7be5c8f52dd597add3cff0cc96c3bf383e1335c9deba0c1ab8e10dfa8b0 not found: ID does not exist" Oct 30 19:20:59.873408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.873336 4789 scope.go:122] "RemoveContainer" containerID="972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68" Oct 30 19:20:59.873540 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:20:59.873525 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68\": container with ID starting with 972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68 not found: ID does not exist" containerID="972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68" Oct 30 19:20:59.873577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:20:59.873546 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68"} err="failed to get container status \"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68\": rpc error: code = NotFound desc = could not find container \"972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68\": container with ID starting with 972054e850a335668039955d2c4b58e3a886ac904946414050c1b9dc44239b68 not found: ID does not exist" Oct 30 19:21:00.011468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:00.011384 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:21:00.015417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:00.015388 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ec7ea-openshift-default-56b9b45bdf-bqddc"] Oct 30 19:21:01.669501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.669464 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" path="/var/lib/kubelet/pods/3db188f4-4939-4295-acdf-ee5077b4c1f4/volumes" Oct 30 19:21:01.889632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.889600 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:21:01.890108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.890093 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:01.890179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.890110 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:01.890179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.890140 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:01.890179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.890145 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:01.890293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.890239 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:01.895479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.895457 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:01.947518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.947493 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-905566-dockercfg-rvjsn\"" Oct 30 19:21:01.951185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.951159 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:21:01.964931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.964903 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqst\" (UniqueName: \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:01.965055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.964977 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:01.993070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:01.993048 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-905566\"" Oct 30 19:21:02.066540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.066515 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqst\" (UniqueName: \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:02.066645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.066563 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:02.067278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.067260 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:02.125064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.125041 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqst\" (UniqueName: \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\") pod \"payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:02.208523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.208460 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:02.584923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.584849 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:21:02.592826 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:02.592778 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318bd54a_e1eb_4fd2_81ab_d767efe33c63.slice/crio-fa7b382b7e9a18a694120d7f5a589361f4dbe5f078c91d16399d9be74f0afbc4 WatchSource:0}: Error finding container fa7b382b7e9a18a694120d7f5a589361f4dbe5f078c91d16399d9be74f0afbc4: Status 404 returned error can't find the container with id fa7b382b7e9a18a694120d7f5a589361f4dbe5f078c91d16399d9be74f0afbc4 Oct 30 19:21:02.594993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.594976 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:21:02.878451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:02.878369 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" event={"ID":"318bd54a-e1eb-4fd2-81ab-d767efe33c63","Type":"ContainerStarted","Data":"fa7b382b7e9a18a694120d7f5a589361f4dbe5f078c91d16399d9be74f0afbc4"} Oct 30 19:21:04.265185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:04.265163 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:21:04.280365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:04.280344 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:21:05.048895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:05.048864 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp"] Oct 30 19:21:05.052264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:05.052235 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e383b-gcmjp"] Oct 30 19:21:05.674960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:05.674925 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd35814f-6da4-438b-b0aa-6c2f738a7b9d" path="/var/lib/kubelet/pods/fd35814f-6da4-438b-b0aa-6c2f738a7b9d/volumes" Oct 30 19:21:06.910766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:06.910730 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" event={"ID":"318bd54a-e1eb-4fd2-81ab-d767efe33c63","Type":"ContainerStarted","Data":"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3"} Oct 30 19:21:06.911153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:06.910863 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:07.023332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:07.023285 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" podStartSLOduration=2.436703148 podStartE2EDuration="6.023272499s" totalImagesPullingTime="3.586569351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:07.020874829 +0000 UTC m=+2103.522785318" watchObservedRunningTime="2026-10-30 19:21:07.023272499 +0000 UTC m=+2103.525182985" Oct 30 19:21:17.919849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:17.919763 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:21:18.196026 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:21:18.195987 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:21:18.877876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:18.877836 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:21:18.878079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:18.878058 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" containerID="cri-o://30693127176a17c9146e99562f0f03f3e3adde6f66a9e68a82c2bad8e190f25a" gracePeriod=30 Oct 30 19:21:21.003950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:21.003918 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:21:21.004407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:21.004125 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" containerID="cri-o://fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66" gracePeriod=30 Oct 30 19:21:21.479985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:21.479948 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:21:21.480146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:21.480012 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:21:24.056800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.056740 4789 generic.go:356] "Generic (PLEG): container finished" podID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerID="30693127176a17c9146e99562f0f03f3e3adde6f66a9e68a82c2bad8e190f25a" exitCode=0 Oct 30 19:21:24.057464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.056823 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" event={"ID":"7d51dae9-9c7d-43a0-8618-b0e226c2f517","Type":"ContainerDied","Data":"30693127176a17c9146e99562f0f03f3e3adde6f66a9e68a82c2bad8e190f25a"} Oct 30 19:21:24.268318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.262756 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:21:24.397240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.397195 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:21:24.397879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.397860 4789 cpu_manager.go:414] "containerMap: removing container" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" Oct 30 19:21:24.397922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.397882 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" Oct 30 19:21:24.397999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.397989 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db188f4-4939-4295-acdf-ee5077b4c1f4" containerName="istio-proxy" Oct 30 19:21:24.398038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.398006 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" containerName="istio-proxy" Oct 30 19:21:24.401434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.401413 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.456310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456277 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456347 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456401 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456455 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456491 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456558 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456599 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.456789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.456650 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.457083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.457058 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:24.457384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.457356 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.457719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.457698 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\") pod \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\" (UID: \"7d51dae9-9c7d-43a0-8618-b0e226c2f517\") " Oct 30 19:21:24.458273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.458243 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.458273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.458267 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.458750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.458726 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.459372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.459339 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.459588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.459564 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.464970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.464943 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "kube-api-access-q4xzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:24.465077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.464947 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:21:24.470108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.470082 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:24.470229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.470174 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token" pod "7d51dae9-9c7d-43a0-8618-b0e226c2f517" (UID: "7d51dae9-9c7d-43a0-8618-b0e226c2f517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:24.473138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.472409 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb4a1b\"" Oct 30 19:21:24.478492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.478470 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:21:24.538075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.538055 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb4a1b-dockercfg-nr2qg\"" Oct 30 19:21:24.559848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.559812 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89hq\" (UniqueName: \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.560012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.559968 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.560271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560249 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xzk\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-kube-api-access-q4xzk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560277 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560292 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560306 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560319 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560332 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.560581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.560346 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d51dae9-9c7d-43a0-8618-b0e226c2f517-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:24.661003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.660970 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.661169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.661153 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89hq\" (UniqueName: \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.662016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.661989 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:24.718997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:24.718933 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89hq\" (UniqueName: \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\") pod \"payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:25.015496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.015412 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:25.067173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.067144 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" Oct 30 19:21:25.067685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.067138 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn" event={"ID":"7d51dae9-9c7d-43a0-8618-b0e226c2f517","Type":"ContainerDied","Data":"3cc8d94cb736011cddd71d0fea559f882a00f343f9fb0eb0a6f9cabedb32b095"} Oct 30 19:21:25.067685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.067327 4789 scope.go:122] "RemoveContainer" containerID="30693127176a17c9146e99562f0f03f3e3adde6f66a9e68a82c2bad8e190f25a" Oct 30 19:21:25.225735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.225687 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:21:25.230025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.229997 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-gsrkn"] Oct 30 19:21:25.404950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.404883 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:21:25.411416 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:25.411390 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e106610_849c_40b5_b25a_06f86f86e7d8.slice/crio-0b70f81bcd342643e39b6f6839839a0571a7dcf9cac28a249500af6ae863057d WatchSource:0}: Error finding container 0b70f81bcd342643e39b6f6839839a0571a7dcf9cac28a249500af6ae863057d: Status 404 returned error can't find the container with id 0b70f81bcd342643e39b6f6839839a0571a7dcf9cac28a249500af6ae863057d Oct 30 19:21:25.669857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:25.669770 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d51dae9-9c7d-43a0-8618-b0e226c2f517" path="/var/lib/kubelet/pods/7d51dae9-9c7d-43a0-8618-b0e226c2f517/volumes" Oct 30 19:21:26.076685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.076645 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" event={"ID":"2e106610-849c-40b5-b25a-06f86f86e7d8","Type":"ContainerStarted","Data":"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72"} Oct 30 19:21:26.077129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.076722 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:26.077129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.076752 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" event={"ID":"2e106610-849c-40b5-b25a-06f86f86e7d8","Type":"ContainerStarted","Data":"0b70f81bcd342643e39b6f6839839a0571a7dcf9cac28a249500af6ae863057d"} Oct 30 19:21:26.191095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.191035 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" podStartSLOduration=2.191018074 podStartE2EDuration="2.191018074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:26.190223552 +0000 UTC m=+2122.692134037" watchObservedRunningTime="2026-10-30 19:21:26.191018074 +0000 UTC m=+2122.692928560" Oct 30 19:21:26.267132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.267111 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:21:26.283281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.283228 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.283389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.283299 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.283783 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.283941 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.284025 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.284098 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.284151 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.284271 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.285770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.284317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\") pod \"467bc477-7e67-4846-b5a0-956fff079645\" (UID: \"467bc477-7e67-4846-b5a0-956fff079645\") " Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.283780 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.285563 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.285938 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.286147 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.286381 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.286653 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "kube-api-access-9qnrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.286591 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:26.290937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.288483 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:26.294670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.294641 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo" pod "467bc477-7e67-4846-b5a0-956fff079645" (UID: "467bc477-7e67-4846-b5a0-956fff079645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:21:26.386308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386278 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467bc477-7e67-4846-b5a0-956fff079645-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386306 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386315 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386326 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnrk\" (UniqueName: \"kubernetes.io/projected/467bc477-7e67-4846-b5a0-956fff079645-kube-api-access-9qnrk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386340 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467bc477-7e67-4846-b5a0-956fff079645-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386348 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386356 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386364 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:26.386544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:26.386373 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467bc477-7e67-4846-b5a0-956fff079645-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:27.084199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.084162 4789 generic.go:356] "Generic (PLEG): container finished" podID="467bc477-7e67-4846-b5a0-956fff079645" containerID="fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66" exitCode=0 Oct 30 19:21:27.084663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.084235 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" Oct 30 19:21:27.084663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.084255 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" event={"ID":"467bc477-7e67-4846-b5a0-956fff079645","Type":"ContainerDied","Data":"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66"} Oct 30 19:21:27.084663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.084309 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776" event={"ID":"467bc477-7e67-4846-b5a0-956fff079645","Type":"ContainerDied","Data":"0c74b79b86b8fdb954d4c9936029264b8797249e462852f431916ae3821e2093"} Oct 30 19:21:27.084663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.084336 4789 scope.go:122] "RemoveContainer" containerID="fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66" Oct 30 19:21:27.094968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.094952 4789 scope.go:122] "RemoveContainer" containerID="fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66" Oct 30 19:21:27.095291 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:21:27.095268 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66\": container with ID starting with fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66 not found: ID does not exist" containerID="fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66" Oct 30 19:21:27.095391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.095297 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66"} err="failed to get container status \"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66\": rpc error: code = NotFound desc = could not find container \"fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66\": container with ID starting with fde8a99bfd5f466578fb67b209bead4368fe16eb9bd65c38a7d3ec84d71a8d66 not found: ID does not exist" Oct 30 19:21:27.285835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.285811 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:21:27.332036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.332013 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-d9776"] Oct 30 19:21:27.668217 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:27.668175 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467bc477-7e67-4846-b5a0-956fff079645" path="/var/lib/kubelet/pods/467bc477-7e67-4846-b5a0-956fff079645/volumes" Oct 30 19:21:31.786603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.786569 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:21:31.787104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.787090 4789 cpu_manager.go:414] "containerMap: removing container" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" Oct 30 19:21:31.787104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.787105 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" Oct 30 19:21:31.787238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.787229 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="467bc477-7e67-4846-b5a0-956fff079645" containerName="istio-proxy" Oct 30 19:21:31.792102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.792079 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.842479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.842447 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84dt\" (UniqueName: \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.842585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.842494 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.916236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.908240 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-539084\"" Oct 30 19:21:31.916236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.908534 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-539084-dockercfg-8jdjv\"" Oct 30 19:21:31.940231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.920892 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:21:31.952232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.949457 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84dt\" (UniqueName: \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.952232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.949510 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.952232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.950561 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:31.967232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.967184 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:21:31.973173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:31.972264 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.006501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.006426 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:21:32.011017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.010989 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84dt\" (UniqueName: \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\") pod \"payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:32.050389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.050317 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.050389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.050362 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptzm\" (UniqueName: \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.105772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.105749 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:32.151160 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.151129 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.151394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.151170 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptzm\" (UniqueName: \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.151798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.151776 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.209718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.209685 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptzm\" (UniqueName: \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\") pod \"payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.287700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.287670 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:32.488114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.488080 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:21:32.494937 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:32.494905 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f1f6a2_3248_4d3b_bb35_5f8db8a09eeb.slice/crio-4cecf37c6906b16e7de91621c96a3bb34c9bfc9f6942cef2c1006a209e901396 WatchSource:0}: Error finding container 4cecf37c6906b16e7de91621c96a3bb34c9bfc9f6942cef2c1006a209e901396: Status 404 returned error can't find the container with id 4cecf37c6906b16e7de91621c96a3bb34c9bfc9f6942cef2c1006a209e901396 Oct 30 19:21:32.539653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.539625 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:21:32.544851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.544832 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556468 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556561 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znpd\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556611 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556667 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556710 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556743 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556800 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556840 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.556992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.556894 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.597605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.597270 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-496c5b-openshift-default-dockercfg-lgr7l\"" Oct 30 19:21:32.683232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.679581 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684719 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684776 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684816 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684881 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684926 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.684985 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.685021 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.685111 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znpd\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.685839 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.686084 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.686357 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.686981 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.687326 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.698371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.687467 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.720228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.718473 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.720228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.719141 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:21:32.760265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.757055 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.770229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.765661 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znpd\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\") pod \"e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:32.832871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.832796 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:21:32.888823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:32.888794 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:33.134288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.133408 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" event={"ID":"60e3a351-1caf-459e-93e3-eed2414b6e31","Type":"ContainerStarted","Data":"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd"} Oct 30 19:21:33.134288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.133481 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" event={"ID":"60e3a351-1caf-459e-93e3-eed2414b6e31","Type":"ContainerStarted","Data":"0f4dc7d15d00532b82ba6672fb8987d00d1aa425394884686a963fda812b7b06"} Oct 30 19:21:33.134288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.133682 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:33.135770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.135738 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" event={"ID":"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb","Type":"ContainerStarted","Data":"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996"} Oct 30 19:21:33.135921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.135788 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" event={"ID":"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb","Type":"ContainerStarted","Data":"4cecf37c6906b16e7de91621c96a3bb34c9bfc9f6942cef2c1006a209e901396"} Oct 30 19:21:33.136489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.136428 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:33.245267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.245200 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" podStartSLOduration=2.245186649 podStartE2EDuration="2.245186649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:33.244127824 +0000 UTC m=+2129.746038309" watchObservedRunningTime="2026-10-30 19:21:33.245186649 +0000 UTC m=+2129.747097135" Oct 30 19:21:33.356458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.356392 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" podStartSLOduration=2.356378024 podStartE2EDuration="2.356378024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:33.354348903 +0000 UTC m=+2129.856259393" watchObservedRunningTime="2026-10-30 19:21:33.356378024 +0000 UTC m=+2129.858288509" Oct 30 19:21:33.385941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.385851 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:21:33.402809 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:33.402766 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86dc6b1_d110_4577_821c_9558190efbd4.slice/crio-dc721272af2e6a17a6a0038130c447465842f9d846012945ad758860f94d44dc WatchSource:0}: Error finding container dc721272af2e6a17a6a0038130c447465842f9d846012945ad758860f94d44dc: Status 404 returned error can't find the container with id dc721272af2e6a17a6a0038130c447465842f9d846012945ad758860f94d44dc Oct 30 19:21:33.406277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.405530 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:33.406277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.405615 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:33.406277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:33.405656 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:34.143948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:34.143909 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerStarted","Data":"72caa5b4452751ce38a10cd8d0308e4a4fae926e7b895a8ee862298be6476dd6"} Oct 30 19:21:34.144352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:34.143974 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerStarted","Data":"dc721272af2e6a17a6a0038130c447465842f9d846012945ad758860f94d44dc"} Oct 30 19:21:34.305846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:34.305793 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podStartSLOduration=2.305776825 podStartE2EDuration="2.305776825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:34.304445634 +0000 UTC m=+2130.806356120" watchObservedRunningTime="2026-10-30 19:21:34.305776825 +0000 UTC m=+2130.807687294" Oct 30 19:21:34.889961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:34.889926 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:34.896116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:34.896089 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:35.150360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:35.150267 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:35.151101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:35.151082 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:21:37.086600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:37.086568 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:21:40.669402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.669370 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:21:40.676564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.676543 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.704562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.704536 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlnf\" (UniqueName: \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.704661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.704568 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.727894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.727870 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:21:40.770960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.770937 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-496c5b\"" Oct 30 19:21:40.806779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.806727 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlnf\" (UniqueName: \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.807103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.807079 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.821549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.821518 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:21:40.822328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.822301 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.835231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.832715 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:40.877235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.876596 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlnf\" (UniqueName: \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:40.897252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.885448 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-496c5b-dockercfg-zfqns\"" Oct 30 19:21:40.897252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.888091 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:21:40.910338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.910304 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:40.910501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.910388 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xln2k\" (UniqueName: \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:40.990827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:40.990802 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:41.011815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.011791 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:41.011943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.011844 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xln2k\" (UniqueName: \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:41.012704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.012659 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:41.075248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.073563 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xln2k\" (UniqueName: \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\") pod \"payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:41.161051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.161019 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:41.377624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.377531 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:21:41.384256 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:41.384228 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c45075f_87ac_46b6_9740_15d889971f7a.slice/crio-e20149c9a92e0d9f8cd729174060d68ccec4436ff18456a4462579819ab0e3fc WatchSource:0}: Error finding container e20149c9a92e0d9f8cd729174060d68ccec4436ff18456a4462579819ab0e3fc: Status 404 returned error can't find the container with id e20149c9a92e0d9f8cd729174060d68ccec4436ff18456a4462579819ab0e3fc Oct 30 19:21:41.538758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:41.538736 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:21:41.544917 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:41.544894 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf6a1fd_fc09_42fd_afe4_635efdfcd52d.slice/crio-70a9817927dea22de22cdad5df988202bf4b6501912f2bba8a36130a288a4235 WatchSource:0}: Error finding container 70a9817927dea22de22cdad5df988202bf4b6501912f2bba8a36130a288a4235: Status 404 returned error can't find the container with id 70a9817927dea22de22cdad5df988202bf4b6501912f2bba8a36130a288a4235 Oct 30 19:21:42.202547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.202509 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" event={"ID":"8c45075f-87ac-46b6-9740-15d889971f7a","Type":"ContainerStarted","Data":"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813"} Oct 30 19:21:42.202983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.202604 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:42.202983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.202632 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" event={"ID":"8c45075f-87ac-46b6-9740-15d889971f7a","Type":"ContainerStarted","Data":"e20149c9a92e0d9f8cd729174060d68ccec4436ff18456a4462579819ab0e3fc"} Oct 30 19:21:42.203973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.203947 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" event={"ID":"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d","Type":"ContainerStarted","Data":"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848"} Oct 30 19:21:42.204065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.203991 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" event={"ID":"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d","Type":"ContainerStarted","Data":"70a9817927dea22de22cdad5df988202bf4b6501912f2bba8a36130a288a4235"} Oct 30 19:21:42.204065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.204012 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:42.314309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.314257 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" podStartSLOduration=2.31424526 podStartE2EDuration="2.31424526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:42.311878679 +0000 UTC m=+2138.813789166" watchObservedRunningTime="2026-10-30 19:21:42.31424526 +0000 UTC m=+2138.816155743" Oct 30 19:21:42.424610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:42.424558 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" podStartSLOduration=2.424540278 podStartE2EDuration="2.424540278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:42.421055995 +0000 UTC m=+2138.922966482" watchObservedRunningTime="2026-10-30 19:21:42.424540278 +0000 UTC m=+2138.926450765" Oct 30 19:21:44.148102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:44.148073 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:21:45.153549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:45.153521 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:21:48.196126 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:21:48.196084 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:21:53.213786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:53.213758 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:21:53.214149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:53.213828 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:21:57.818611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.818573 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:21:57.823641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.823606 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.877474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.877445 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d6b5ac-openshift-default-dockercfg-mxfgc\"" Oct 30 19:21:57.936931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.936893 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:21:57.937692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937662 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937706 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937746 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937778 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937844 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937863 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937890 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937905 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:57.937986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:57.937931 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2pn\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.052293 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054319 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054419 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054480 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054649 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054693 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054721 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.054799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.054763 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2pn\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.060240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.055540 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.060240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.056046 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.060240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.056699 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.079280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.064592 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.079280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.065020 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.083072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.083038 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.131279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.111097 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.131549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.116258 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.141262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.141199 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2pn\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\") pod \"e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.439734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.439701 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:58.887005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.886931 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:21:58.893692 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:21:58.893658 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cce3d0_2210_4a42_8c43_970429d4e7b1.slice/crio-ed03ed9b9db15e08ec34dd79a4e0fd66fbfa439ac619b44774967a27d10031d3 WatchSource:0}: Error finding container ed03ed9b9db15e08ec34dd79a4e0fd66fbfa439ac619b44774967a27d10031d3: Status 404 returned error can't find the container with id ed03ed9b9db15e08ec34dd79a4e0fd66fbfa439ac619b44774967a27d10031d3 Oct 30 19:21:58.897182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.897144 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.897437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.897409 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:58.897505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:58.897480 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:21:59.323472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:59.323417 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerStarted","Data":"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859"} Oct 30 19:21:59.323673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:59.323548 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerStarted","Data":"ed03ed9b9db15e08ec34dd79a4e0fd66fbfa439ac619b44774967a27d10031d3"} Oct 30 19:21:59.440032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:59.440001 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:59.445993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:59.445964 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:21:59.486059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:21:59.486014 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podStartSLOduration=2.486001931 podStartE2EDuration="2.486001931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:59.485629644 +0000 UTC m=+2155.987540130" watchObservedRunningTime="2026-10-30 19:21:59.486001931 +0000 UTC m=+2155.987912417" Oct 30 19:22:00.330567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:00.330532 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:22:00.331494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:00.331472 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:22:05.736185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:05.736154 4789 scope.go:122] "RemoveContainer" containerID="bbb974371f7ae07a0ab172da6393794a79a9edfb3095f95e8eb719715d5cda0c" Oct 30 19:22:05.746551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:05.746517 4789 scope.go:122] "RemoveContainer" containerID="ff60d144a4ce0de01b42aea8cab13846220c0a138c488d4611dfbb2f8bd22c2b" Oct 30 19:22:05.755519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:05.755502 4789 scope.go:122] "RemoveContainer" containerID="06121218b32947cd370426e23395b315dab9089fc7f1c4290094a903c7bd180c" Oct 30 19:22:06.703833 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.703801 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:22:06.712428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.711676 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.766534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.765846 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:22:06.775469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.775432 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvvp\" (UniqueName: \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.775644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.775483 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.782919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.782886 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d6b5ac\"" Oct 30 19:22:06.806841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.806817 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d6b5ac-dockercfg-wh7jg\"" Oct 30 19:22:06.877072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.877032 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvvp\" (UniqueName: \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.877308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.877103 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.878831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.878316 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:06.938498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:06.938461 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvvp\" (UniqueName: \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\") pod \"payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:07.061731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:07.061659 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:07.446538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:07.446509 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:22:07.459460 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:22:07.459384 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfb9d87_7a69_480b_a9ad_ab8ca2351164.slice/crio-f1f1d2c1abdd68d382c086fce6e2aebbd9a9e85dd3266295ac4478e88bc6c439 WatchSource:0}: Error finding container f1f1d2c1abdd68d382c086fce6e2aebbd9a9e85dd3266295ac4478e88bc6c439: Status 404 returned error can't find the container with id f1f1d2c1abdd68d382c086fce6e2aebbd9a9e85dd3266295ac4478e88bc6c439 Oct 30 19:22:08.389726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:08.389687 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" event={"ID":"abfb9d87-7a69-480b-a9ad-ab8ca2351164","Type":"ContainerStarted","Data":"1b83e0ab4ba451d444ca7c127773daf0165099379e9f055dbd9ca66cb139b063"} Oct 30 19:22:08.390131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:08.389844 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:08.390131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:08.389891 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" event={"ID":"abfb9d87-7a69-480b-a9ad-ab8ca2351164","Type":"ContainerStarted","Data":"f1f1d2c1abdd68d382c086fce6e2aebbd9a9e85dd3266295ac4478e88bc6c439"} Oct 30 19:22:08.512793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:08.512742 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" podStartSLOduration=2.512729052 podStartE2EDuration="2.512729052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:08.510164059 +0000 UTC m=+2165.012074547" watchObservedRunningTime="2026-10-30 19:22:08.512729052 +0000 UTC m=+2165.014639538" Oct 30 19:22:18.197073 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:18.197035 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:22:18.941790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:18.941758 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:22:18.941997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:18.941958 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" containerID="cri-o://1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e" gracePeriod=30 Oct 30 19:22:19.400778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:19.400696 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:22:21.570231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:21.570180 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:22:21.570688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:21.570410 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" containerID="cri-o://3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f" gracePeriod=30 Oct 30 19:22:23.447086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:23.447050 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:23.447514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:23.447120 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:23.672393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:23.672355 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:23.672576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:23.672429 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:24.218407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.218346 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:22:24.391140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391109 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391161 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391238 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391302 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391414 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:24.391678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391486 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391518 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391559 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:24.391678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391576 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391676 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:24.391902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391680 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\") pod \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\" (UID: \"4f5f4586-0772-4c45-8ba7-68cb4585ad78\") " Oct 30 19:22:24.391902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391775 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:24.391902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.391696 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:24.392433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.392303 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.392433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.392328 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.392433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.392342 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.392433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.392355 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.392433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.392368 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.393919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.393896 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:24.394107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.394089 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:24.394178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.394121 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:24.394311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.394292 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8" pod "4f5f4586-0772-4c45-8ba7-68cb4585ad78" (UID: "4f5f4586-0772-4c45-8ba7-68cb4585ad78"). InnerVolumeSpecName "kube-api-access-9stk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:24.493434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.493341 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.493434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.493373 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9stk8\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-kube-api-access-9stk8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.493434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.493388 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.493434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.493401 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5f4586-0772-4c45-8ba7-68cb4585ad78-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:24.506870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.506826 4789 generic.go:356] "Generic (PLEG): container finished" podID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerID="1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e" exitCode=0 Oct 30 19:22:24.507052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.506881 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" Oct 30 19:22:24.507052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.506887 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" event={"ID":"4f5f4586-0772-4c45-8ba7-68cb4585ad78","Type":"ContainerDied","Data":"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e"} Oct 30 19:22:24.507052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.506954 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5" event={"ID":"4f5f4586-0772-4c45-8ba7-68cb4585ad78","Type":"ContainerDied","Data":"ae53fb43950da1d6fde0484bbbf074cbe54f74fa7d9e505d62eb87696e46248d"} Oct 30 19:22:24.507052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.506985 4789 scope.go:122] "RemoveContainer" containerID="1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e" Oct 30 19:22:24.519625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.519602 4789 scope.go:122] "RemoveContainer" containerID="1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e" Oct 30 19:22:24.519919 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:24.519899 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e\": container with ID starting with 1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e not found: ID does not exist" containerID="1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e" Oct 30 19:22:24.519996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.519930 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e"} err="failed to get container status \"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e\": rpc error: code = NotFound desc = could not find container \"1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e\": container with ID starting with 1785381351e2da2698aaae8ba714dc86e901325f3be1bfa2fda311c6f409e27e not found: ID does not exist" Oct 30 19:22:24.663822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.663787 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:22:24.667814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:24.667786 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-n6qq5"] Oct 30 19:22:25.668890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:25.668853 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" path="/var/lib/kubelet/pods/4f5f4586-0772-4c45-8ba7-68cb4585ad78/volumes" Oct 30 19:22:26.849191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.849122 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:22:26.919646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919613 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.919793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919685 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.919793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919733 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.919793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919781 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.919943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919826 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.919943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919856 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.920050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919953 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:26.920317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.919990 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.920459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920359 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.920459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920030 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:26.920459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920440 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\") pod \"0afc72ed-573a-4879-9388-4c6d208c953e\" (UID: \"0afc72ed-573a-4879-9388-4c6d208c953e\") " Oct 30 19:22:26.920649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920255 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:26.920649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920285 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:26.920649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920568 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:26.921013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.920992 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.921090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.921022 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.921090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.921041 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.921090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.921055 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0afc72ed-573a-4879-9388-4c6d208c953e-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.921090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.921070 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:26.922350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.922321 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:26.922549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.922526 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:26.922624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.922574 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:26.922662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:26.922636 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d" pod "0afc72ed-573a-4879-9388-4c6d208c953e" (UID: "0afc72ed-573a-4879-9388-4c6d208c953e"). InnerVolumeSpecName "kube-api-access-6hx9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:27.022281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.022248 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0afc72ed-573a-4879-9388-4c6d208c953e-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:27.022281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.022280 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:27.022485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.022292 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0afc72ed-573a-4879-9388-4c6d208c953e-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:27.022485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.022306 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hx9d\" (UniqueName: \"kubernetes.io/projected/0afc72ed-573a-4879-9388-4c6d208c953e-kube-api-access-6hx9d\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:27.528078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.528045 4789 generic.go:356] "Generic (PLEG): container finished" podID="0afc72ed-573a-4879-9388-4c6d208c953e" containerID="3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f" exitCode=0 Oct 30 19:22:27.528297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.528093 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" Oct 30 19:22:27.528297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.528132 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" event={"ID":"0afc72ed-573a-4879-9388-4c6d208c953e","Type":"ContainerDied","Data":"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f"} Oct 30 19:22:27.528297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.528186 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg" event={"ID":"0afc72ed-573a-4879-9388-4c6d208c953e","Type":"ContainerDied","Data":"61df095c45a198706ee67b7f3a20efbac1dde601408264d30614b05dacd7ca2a"} Oct 30 19:22:27.528297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.528197 4789 scope.go:122] "RemoveContainer" containerID="3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f" Oct 30 19:22:27.549125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.549101 4789 scope.go:122] "RemoveContainer" containerID="3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f" Oct 30 19:22:27.549460 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:27.549440 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f\": container with ID starting with 3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f not found: ID does not exist" containerID="3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f" Oct 30 19:22:27.549521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.549472 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f"} err="failed to get container status \"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f\": rpc error: code = NotFound desc = could not find container \"3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f\": container with ID starting with 3887f4e4a9637cafddc497d55537448c7ccae98942b5a185a9912e8dcbad339f not found: ID does not exist" Oct 30 19:22:27.730779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.730746 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:22:27.778737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:27.778661 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-qh7vg"] Oct 30 19:22:29.668661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:29.668629 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" path="/var/lib/kubelet/pods/0afc72ed-573a-4879-9388-4c6d208c953e/volumes" Oct 30 19:22:35.497046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:35.497014 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:22:35.497537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:35.497246 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" containerID="cri-o://8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d" gracePeriod=30 Oct 30 19:22:36.693059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:36.693019 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:22:36.693558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:36.693279 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" containerID="cri-o://ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb" gracePeriod=30 Oct 30 19:22:40.120680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.120642 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:40.121057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.120715 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:40.778801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.778739 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:22:40.877638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877600 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.877807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877669 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.877807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877689 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.877807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877716 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.877807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877799 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877816 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877899 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877915 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877940 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.877988 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:40.878040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878001 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:40.878406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878044 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\") pod \"5fec5d34-5588-4f06-96ad-447feba5391b\" (UID: \"5fec5d34-5588-4f06-96ad-447feba5391b\") " Oct 30 19:22:40.878406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878067 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:40.878406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878158 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:40.878572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878495 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.878572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878516 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.878572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878530 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.878572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878542 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.878572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.878557 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fec5d34-5588-4f06-96ad-447feba5391b-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.880177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.880154 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:40.880298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.880182 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "kube-api-access-fqv4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:40.880298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.880185 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:40.880298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.880240 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo" pod "5fec5d34-5588-4f06-96ad-447feba5391b" (UID: "5fec5d34-5588-4f06-96ad-447feba5391b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:40.980168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.980141 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqv4t\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-kube-api-access-fqv4t\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.980168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.980166 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fec5d34-5588-4f06-96ad-447feba5391b-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.980390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.980176 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fec5d34-5588-4f06-96ad-447feba5391b-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:40.980390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:40.980191 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fec5d34-5588-4f06-96ad-447feba5391b-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:41.141088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.141005 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:41.141088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.141073 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:41.619666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.619631 4789 generic.go:356] "Generic (PLEG): container finished" podID="5fec5d34-5588-4f06-96ad-447feba5391b" containerID="8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d" exitCode=0 Oct 30 19:22:41.619855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.619681 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" Oct 30 19:22:41.619855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.619707 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" event={"ID":"5fec5d34-5588-4f06-96ad-447feba5391b","Type":"ContainerDied","Data":"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d"} Oct 30 19:22:41.619855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.619770 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7" event={"ID":"5fec5d34-5588-4f06-96ad-447feba5391b","Type":"ContainerDied","Data":"35a8a76c73e8d38a717336579b18d07d948fc1e539be46b35411b903b5090540"} Oct 30 19:22:41.619855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.619816 4789 scope.go:122] "RemoveContainer" containerID="8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d" Oct 30 19:22:41.631408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.631315 4789 scope.go:122] "RemoveContainer" containerID="8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d" Oct 30 19:22:41.631614 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:41.631585 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d\": container with ID starting with 8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d not found: ID does not exist" containerID="8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d" Oct 30 19:22:41.631680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.631622 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d"} err="failed to get container status \"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d\": rpc error: code = NotFound desc = could not find container \"8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d\": container with ID starting with 8ee80db20ea57c781bd3e7f038712e2e190efb85582e4f3aeb8c69570c182b4d not found: ID does not exist" Oct 30 19:22:41.775470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.775433 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:22:41.824145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.824119 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-wvls7"] Oct 30 19:22:41.974511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:41.974487 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:22:42.094709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.094672 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.095069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.095044 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.095493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.095471 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.095889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.095870 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.096250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096231 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.096575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096559 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.096716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096701 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.097267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.097249 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.097718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.097700 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\") pod \"98cff649-d727-4d15-88a9-3fabb4833514\" (UID: \"98cff649-d727-4d15-88a9-3fabb4833514\") " Oct 30 19:22:42.105600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096032 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:42.105600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096373 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:42.105600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.096857 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:42.105600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.097402 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:42.107396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.107361 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:42.111780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.111749 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:42.113951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.113931 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:42.114432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.114025 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:42.115314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.115289 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h" pod "98cff649-d727-4d15-88a9-3fabb4833514" (UID: "98cff649-d727-4d15-88a9-3fabb4833514"). InnerVolumeSpecName "kube-api-access-l677h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.199997 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cff649-d727-4d15-88a9-3fabb4833514-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200035 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200051 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200066 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cff649-d727-4d15-88a9-3fabb4833514-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200078 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l677h\" (UniqueName: \"kubernetes.io/projected/98cff649-d727-4d15-88a9-3fabb4833514-kube-api-access-l677h\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200093 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200107 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200120 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.200414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.200133 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cff649-d727-4d15-88a9-3fabb4833514-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:42.627177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.627141 4789 generic.go:356] "Generic (PLEG): container finished" podID="98cff649-d727-4d15-88a9-3fabb4833514" containerID="ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb" exitCode=0 Oct 30 19:22:42.627395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.627200 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" Oct 30 19:22:42.627395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.627233 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" event={"ID":"98cff649-d727-4d15-88a9-3fabb4833514","Type":"ContainerDied","Data":"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb"} Oct 30 19:22:42.627395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.627327 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch" event={"ID":"98cff649-d727-4d15-88a9-3fabb4833514","Type":"ContainerDied","Data":"7d74e5f700f5c9e6291447fcde9ab214d0973c12742d342ec76e4a29137cf688"} Oct 30 19:22:42.627395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.627362 4789 scope.go:122] "RemoveContainer" containerID="ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb" Oct 30 19:22:42.637004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.636982 4789 scope.go:122] "RemoveContainer" containerID="ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb" Oct 30 19:22:42.637298 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:42.637276 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb\": container with ID starting with ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb not found: ID does not exist" containerID="ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb" Oct 30 19:22:42.637395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.637303 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb"} err="failed to get container status \"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb\": rpc error: code = NotFound desc = could not find container \"ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb\": container with ID starting with ff30275aec3e5911b545e3a68094564f40eea87a7197afaaf5f8d5119be0a2eb not found: ID does not exist" Oct 30 19:22:42.782522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.782491 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:22:42.829054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:42.829021 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-c6zch"] Oct 30 19:22:43.668337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:43.668304 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" path="/var/lib/kubelet/pods/5fec5d34-5588-4f06-96ad-447feba5391b/volumes" Oct 30 19:22:43.668789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:43.668775 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cff649-d727-4d15-88a9-3fabb4833514" path="/var/lib/kubelet/pods/98cff649-d727-4d15-88a9-3fabb4833514/volumes" Oct 30 19:22:46.050269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:46.050232 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5"] Oct 30 19:22:46.096426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:46.096384 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8052e80-mxsq5"] Oct 30 19:22:47.667512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:22:47.667481 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef638095-80ad-43ea-ad32-29019f8b6c5a" path="/var/lib/kubelet/pods/ef638095-80ad-43ea-ad32-29019f8b6c5a/volumes" Oct 30 19:22:48.198468 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:22:48.198425 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:23:05.800544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:05.800510 4789 scope.go:122] "RemoveContainer" containerID="e27abf0bc51aed910e4b05f4117048f57e86fe61b8473b9b317507d88f9b2531" Oct 30 19:23:05.811044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:05.811014 4789 scope.go:122] "RemoveContainer" containerID="d4bbd2b6c3be66cebcfc844107143c5c44633a107896d93860b3e8fa3b432de0" Oct 30 19:23:06.674008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.673971 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw"] Oct 30 19:23:06.674589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674568 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674589 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674611 4789 cpu_manager.go:414] "containerMap: removing container" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674618 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674646 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674654 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674681 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" Oct 30 19:23:06.674711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674688 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" Oct 30 19:23:06.675054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674844 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fec5d34-5588-4f06-96ad-447feba5391b" containerName="istio-proxy" Oct 30 19:23:06.675054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674860 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cff649-d727-4d15-88a9-3fabb4833514" containerName="istio-proxy" Oct 30 19:23:06.675054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674874 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5f4586-0772-4c45-8ba7-68cb4585ad78" containerName="istio-proxy" Oct 30 19:23:06.675054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.674886 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afc72ed-573a-4879-9388-4c6d208c953e" containerName="istio-proxy" Oct 30 19:23:06.680151 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.680130 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.730154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.730130 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:23:06.775385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.775351 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2l9r\" (UniqueName: \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.775548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.775428 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.775624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.775580 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.776172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.776151 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:06.776252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.776221 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:23:06.776252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.776239 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:23:06.778999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.778980 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw"] Oct 30 19:23:06.876888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.876851 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.877304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.876919 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2l9r\" (UniqueName: \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.877304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.876968 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.877508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.877486 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.879376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.879358 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.934746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.934720 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2l9r\" (UniqueName: \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:06.993343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:06.993318 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:07.392415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:07.392323 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw"] Oct 30 19:23:07.399065 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:23:07.399021 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c97deaa_ece4_4240_9e3c_7686b39be98f.slice/crio-b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6 WatchSource:0}: Error finding container b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6: Status 404 returned error can't find the container with id b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6 Oct 30 19:23:07.790812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:07.790779 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" event={"ID":"7c97deaa-ece4-4240-9e3c-7686b39be98f","Type":"ContainerStarted","Data":"3257e49fd12449b0b652c56d6928b87ae527b9ab150fa7bd6ad06abd0290888a"} Oct 30 19:23:07.790930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:07.790832 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" event={"ID":"7c97deaa-ece4-4240-9e3c-7686b39be98f","Type":"ContainerStarted","Data":"b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6"} Oct 30 19:23:07.906734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:07.906657 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" podStartSLOduration=1.906638728 podStartE2EDuration="1.906638728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:07.90005756 +0000 UTC m=+2224.401968047" watchObservedRunningTime="2026-10-30 19:23:07.906638728 +0000 UTC m=+2224.408549218" Oct 30 19:23:08.798862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:08.798824 4789 generic.go:356] "Generic (PLEG): container finished" podID="7c97deaa-ece4-4240-9e3c-7686b39be98f" containerID="3257e49fd12449b0b652c56d6928b87ae527b9ab150fa7bd6ad06abd0290888a" exitCode=0 Oct 30 19:23:08.799044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:08.798911 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" event={"ID":"7c97deaa-ece4-4240-9e3c-7686b39be98f","Type":"ContainerDied","Data":"3257e49fd12449b0b652c56d6928b87ae527b9ab150fa7bd6ad06abd0290888a"} Oct 30 19:23:09.980798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:09.980773 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:10.109913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.109813 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\") pod \"7c97deaa-ece4-4240-9e3c-7686b39be98f\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " Oct 30 19:23:10.110120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.109910 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\") pod \"7c97deaa-ece4-4240-9e3c-7686b39be98f\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " Oct 30 19:23:10.110120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.110089 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\" (UniqueName: \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\") pod \"7c97deaa-ece4-4240-9e3c-7686b39be98f\" (UID: \"7c97deaa-ece4-4240-9e3c-7686b39be98f\") " Oct 30 19:23:10.110313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.110284 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca" pod "7c97deaa-ece4-4240-9e3c-7686b39be98f" (UID: "7c97deaa-ece4-4240-9e3c-7686b39be98f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:10.111134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.111111 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:10.112844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.112813 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r" pod "7c97deaa-ece4-4240-9e3c-7686b39be98f" (UID: "7c97deaa-ece4-4240-9e3c-7686b39be98f"). InnerVolumeSpecName "kube-api-access-g2l9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:10.112963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.112879 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca" pod "7c97deaa-ece4-4240-9e3c-7686b39be98f" (UID: "7c97deaa-ece4-4240-9e3c-7686b39be98f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:10.212585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.212552 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2l9r\" (UniqueName: \"kubernetes.io/projected/7c97deaa-ece4-4240-9e3c-7686b39be98f-kube-api-access-g2l9r\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:10.212585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.212581 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c97deaa-ece4-4240-9e3c-7686b39be98f-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:10.819309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.819271 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" event={"ID":"7c97deaa-ece4-4240-9e3c-7686b39be98f","Type":"ContainerDied","Data":"b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6"} Oct 30 19:23:10.819524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.819323 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7efb2fa64b6b0603fd14aad67f256c3135748f4370e1494fecae7fe5c623de6" Oct 30 19:23:10.819524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:10.819289 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw" Oct 30 19:23:18.196754 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:23:18.196713 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:23:21.193392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.193347 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:23:21.193820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.193436 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:23:21.908780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.908698 4789 generic.go:356] "Generic (PLEG): container finished" podID="e86dc6b1-d110-4577-821c-9558190efbd4" containerID="72caa5b4452751ce38a10cd8d0308e4a4fae926e7b895a8ee862298be6476dd6" exitCode=0 Oct 30 19:23:21.908969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.908771 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerDied","Data":"72caa5b4452751ce38a10cd8d0308e4a4fae926e7b895a8ee862298be6476dd6"} Oct 30 19:23:21.909182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.909167 4789 scope.go:122] "RemoveContainer" containerID="72caa5b4452751ce38a10cd8d0308e4a4fae926e7b895a8ee862298be6476dd6" Oct 30 19:23:21.910739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.910713 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:21.910807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.910782 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:21.910853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:21.910814 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:22.933304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:22.933270 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerStarted","Data":"3c52e5d8c82a782fe8732443d112b383e45046b3c455eb35bae3493de65599b8"} Oct 30 19:23:23.889380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:23.889351 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:23:23.889576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:23.889512 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:23:23.889576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:23.889566 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 19:23:24.889494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:24.889459 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:23:24.889888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:24.889533 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 19:23:25.889362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:25.889324 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:23:25.889542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:25.889387 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 19:23:26.890011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:26.889980 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:23:26.890502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:26.890038 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 19:23:27.889564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:27.889528 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Oct 30 19:23:27.889735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:27.889596 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Oct 30 19:23:28.895533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:28.895498 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:23:28.896002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:28.895937 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:23:28.896722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:28.896703 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:23:39.587512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:39.587476 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:23:39.587951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:39.587692 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" podUID="2e106610-849c-40b5-b25a-06f86f86e7d8" containerName="payload-pre-processing" containerID="cri-o://6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72" gracePeriod=30 Oct 30 19:23:39.903133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:39.903105 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.070731 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\") pod \"2e106610-849c-40b5-b25a-06f86f86e7d8\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.070838 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\" (UniqueName: \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\") pod \"2e106610-849c-40b5-b25a-06f86f86e7d8\" (UID: \"2e106610-849c-40b5-b25a-06f86f86e7d8\") " Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.073881 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume" pod "2e106610-849c-40b5-b25a-06f86f86e7d8" (UID: "2e106610-849c-40b5-b25a-06f86f86e7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.082769 4789 generic.go:356] "Generic (PLEG): container finished" podID="2e106610-849c-40b5-b25a-06f86f86e7d8" containerID="6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72" exitCode=0 Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.082960 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" event={"ID":"2e106610-849c-40b5-b25a-06f86f86e7d8","Type":"ContainerDied","Data":"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72"} Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.083010 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" event={"ID":"2e106610-849c-40b5-b25a-06f86f86e7d8","Type":"ContainerDied","Data":"0b70f81bcd342643e39b6f6839839a0571a7dcf9cac28a249500af6ae863057d"} Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.083042 4789 scope.go:122] "RemoveContainer" containerID="6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72" Oct 30 19:23:40.087253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.083195 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw" Oct 30 19:23:40.087926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.087830 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq" pod "2e106610-849c-40b5-b25a-06f86f86e7d8" (UID: "2e106610-849c-40b5-b25a-06f86f86e7d8"). InnerVolumeSpecName "kube-api-access-b89hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:40.114414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.114282 4789 scope.go:122] "RemoveContainer" containerID="6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72" Oct 30 19:23:40.115232 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:23:40.115039 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72\": container with ID starting with 6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72 not found: ID does not exist" containerID="6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72" Oct 30 19:23:40.115232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.115078 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72"} err="failed to get container status \"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72\": rpc error: code = NotFound desc = could not find container \"6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72\": container with ID starting with 6696f384f6bf1370a274e411b43eb294021f553909d9b3b01407461e42e69a72 not found: ID does not exist" Oct 30 19:23:40.172782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.172700 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e106610-849c-40b5-b25a-06f86f86e7d8-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:40.172782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.172730 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89hq\" (UniqueName: \"kubernetes.io/projected/2e106610-849c-40b5-b25a-06f86f86e7d8-kube-api-access-b89hq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:40.504594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.503358 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:23:40.548116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:40.548077 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fb4a1b-9b9c844ff-xngkw"] Oct 30 19:23:41.301000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:41.300947 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw"] Oct 30 19:23:41.305007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:41.304963 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb4a1b-5rjpw"] Oct 30 19:23:41.669108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:41.669020 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e106610-849c-40b5-b25a-06f86f86e7d8" path="/var/lib/kubelet/pods/2e106610-849c-40b5-b25a-06f86f86e7d8/volumes" Oct 30 19:23:41.669620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:41.669599 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c97deaa-ece4-4240-9e3c-7686b39be98f" path="/var/lib/kubelet/pods/7c97deaa-ece4-4240-9e3c-7686b39be98f/volumes" Oct 30 19:23:43.106767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.106722 4789 generic.go:356] "Generic (PLEG): container finished" podID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerID="bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859" exitCode=0 Oct 30 19:23:43.107577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.106794 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerDied","Data":"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859"} Oct 30 19:23:43.107577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.107427 4789 scope.go:122] "RemoveContainer" containerID="bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859" Oct 30 19:23:43.108994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.108965 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:43.109089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.109035 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:43.109089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:43.109071 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:23:44.130176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:44.130144 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerStarted","Data":"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360"} Oct 30 19:23:44.439957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:44.439921 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:23:44.440916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:44.440889 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:44.441041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:44.440942 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:45.331497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:45.331465 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:23:45.440445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:45.440414 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:45.440655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:45.440478 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:46.440157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:46.440110 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:46.440706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:46.440165 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:47.440701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:47.440664 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:47.441199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:47.440728 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:48.227588 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:23:48.227536 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:23:48.440671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:48.440636 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:48.440936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:48.440906 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:49.440621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.440583 4789 patch_prober.go:29] interesting pod/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Oct 30 19:23:49.440826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.440655 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Oct 30 19:23:49.756567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.756486 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf"] Oct 30 19:23:49.757204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757181 4789 cpu_manager.go:414] "containerMap: removing container" podUID="7c97deaa-ece4-4240-9e3c-7686b39be98f" containerName="revoke-keys" Oct 30 19:23:49.757204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757220 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c97deaa-ece4-4240-9e3c-7686b39be98f" containerName="revoke-keys" Oct 30 19:23:49.757387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757260 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2e106610-849c-40b5-b25a-06f86f86e7d8" containerName="payload-pre-processing" Oct 30 19:23:49.757387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757269 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e106610-849c-40b5-b25a-06f86f86e7d8" containerName="payload-pre-processing" Oct 30 19:23:49.757493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757430 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c97deaa-ece4-4240-9e3c-7686b39be98f" containerName="revoke-keys" Oct 30 19:23:49.757493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.757450 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e106610-849c-40b5-b25a-06f86f86e7d8" containerName="payload-pre-processing" Oct 30 19:23:49.760221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.760190 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.795672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.795648 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.795802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.795696 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5rr\" (UniqueName: \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.795802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.795731 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.816225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.816075 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf"] Oct 30 19:23:49.859277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.859250 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:23:49.859415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.859250 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:23:49.859415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.859262 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:23:49.859415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.859353 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:23:49.896950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.896922 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.897070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.896959 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5rr\" (UniqueName: \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.897070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.896982 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.897710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.897690 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.899326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.899307 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:49.954453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:49.954434 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5rr\" (UniqueName: \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\") pod \"maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:50.073191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:50.073110 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:50.448712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:50.448683 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:23:50.449880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:50.449858 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:23:50.449992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:50.449910 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf"] Oct 30 19:23:50.455165 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:23:50.455139 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88dceeb4_1e61_43ed_9fe8_d76921458c82.slice/crio-9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c WatchSource:0}: Error finding container 9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c: Status 404 returned error can't find the container with id 9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c Oct 30 19:23:51.178144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:51.178058 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" event={"ID":"88dceeb4-1e61-43ed-9fe8-d76921458c82","Type":"ContainerStarted","Data":"84da86575171ab170bc63fec5ecdcb489a7da17de16d17b0e0346a850d2da913"} Oct 30 19:23:51.178144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:51.178119 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" event={"ID":"88dceeb4-1e61-43ed-9fe8-d76921458c82","Type":"ContainerStarted","Data":"9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c"} Oct 30 19:23:51.287798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:51.287750 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" podStartSLOduration=2.287733562 podStartE2EDuration="2.287733562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:51.287143319 +0000 UTC m=+2267.789053806" watchObservedRunningTime="2026-10-30 19:23:51.287733562 +0000 UTC m=+2267.789644049" Oct 30 19:23:52.186459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:52.186426 4789 generic.go:356] "Generic (PLEG): container finished" podID="88dceeb4-1e61-43ed-9fe8-d76921458c82" containerID="84da86575171ab170bc63fec5ecdcb489a7da17de16d17b0e0346a850d2da913" exitCode=0 Oct 30 19:23:52.186843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:52.186504 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" event={"ID":"88dceeb4-1e61-43ed-9fe8-d76921458c82","Type":"ContainerDied","Data":"84da86575171ab170bc63fec5ecdcb489a7da17de16d17b0e0346a850d2da913"} Oct 30 19:23:53.364533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.364507 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:53.438406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.438377 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\") pod \"88dceeb4-1e61-43ed-9fe8-d76921458c82\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " Oct 30 19:23:53.438562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.438429 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\") pod \"88dceeb4-1e61-43ed-9fe8-d76921458c82\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " Oct 30 19:23:53.438562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.438477 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\" (UniqueName: \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\") pod \"88dceeb4-1e61-43ed-9fe8-d76921458c82\" (UID: \"88dceeb4-1e61-43ed-9fe8-d76921458c82\") " Oct 30 19:23:53.438814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.438782 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca" pod "88dceeb4-1e61-43ed-9fe8-d76921458c82" (UID: "88dceeb4-1e61-43ed-9fe8-d76921458c82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:53.438938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.438920 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:53.440545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.440517 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca" pod "88dceeb4-1e61-43ed-9fe8-d76921458c82" (UID: "88dceeb4-1e61-43ed-9fe8-d76921458c82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:53.440825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.440806 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr" pod "88dceeb4-1e61-43ed-9fe8-d76921458c82" (UID: "88dceeb4-1e61-43ed-9fe8-d76921458c82"). InnerVolumeSpecName "kube-api-access-hq5rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:53.539988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.539957 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88dceeb4-1e61-43ed-9fe8-d76921458c82-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:53.539988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:53.539988 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5rr\" (UniqueName: \"kubernetes.io/projected/88dceeb4-1e61-43ed-9fe8-d76921458c82-kube-api-access-hq5rr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:54.199811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:54.199787 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" Oct 30 19:23:54.199811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:54.199799 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf" event={"ID":"88dceeb4-1e61-43ed-9fe8-d76921458c82","Type":"ContainerDied","Data":"9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c"} Oct 30 19:23:54.200022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:54.199844 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9350af751b9470ceb9cc4000f80fe089c89347fe4835cf5042b96e32c69a7c7c" Oct 30 19:23:56.342958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:56.342921 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:23:56.441616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:23:56.441584 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:24:01.343526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.343469 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" podUID="60e3a351-1caf-459e-93e3-eed2414b6e31" containerName="payload-processing" containerID="cri-o://d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd" gracePeriod=25 Oct 30 19:24:01.441884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.441824 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" podUID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" containerName="payload-pre-processing" containerID="cri-o://6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996" gracePeriod=25 Oct 30 19:24:01.639118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.638759 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:24:01.735004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.734972 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\") pod \"60e3a351-1caf-459e-93e3-eed2414b6e31\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " Oct 30 19:24:01.735187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.735128 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\" (UniqueName: \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\") pod \"60e3a351-1caf-459e-93e3-eed2414b6e31\" (UID: \"60e3a351-1caf-459e-93e3-eed2414b6e31\") " Oct 30 19:24:01.736173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.736139 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume" pod "60e3a351-1caf-459e-93e3-eed2414b6e31" (UID: "60e3a351-1caf-459e-93e3-eed2414b6e31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:01.737805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.737780 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm" pod "60e3a351-1caf-459e-93e3-eed2414b6e31" (UID: "60e3a351-1caf-459e-93e3-eed2414b6e31"). InnerVolumeSpecName "kube-api-access-wptzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:01.760100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.760073 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:24:01.837070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.837034 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptzm\" (UniqueName: \"kubernetes.io/projected/60e3a351-1caf-459e-93e3-eed2414b6e31-kube-api-access-wptzm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:01.837070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.837064 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e3a351-1caf-459e-93e3-eed2414b6e31-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:01.937838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.937812 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\" (UniqueName: \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\") pod \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " Oct 30 19:24:01.937993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.937937 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\") pod \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\" (UID: \"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb\") " Oct 30 19:24:01.938318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.938298 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume" pod "12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" (UID: "12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:01.939915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:01.939894 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt" pod "12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" (UID: "12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb"). InnerVolumeSpecName "kube-api-access-k84dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:02.039440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.039412 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84dt\" (UniqueName: \"kubernetes.io/projected/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-kube-api-access-k84dt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:02.039440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.039438 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:02.253482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.253387 4789 generic.go:356] "Generic (PLEG): container finished" podID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" containerID="6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996" exitCode=0 Oct 30 19:24:02.253482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.253430 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" Oct 30 19:24:02.253482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.253465 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" event={"ID":"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb","Type":"ContainerDied","Data":"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996"} Oct 30 19:24:02.253756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.253519 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb" event={"ID":"12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb","Type":"ContainerDied","Data":"4cecf37c6906b16e7de91621c96a3bb34c9bfc9f6942cef2c1006a209e901396"} Oct 30 19:24:02.253756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.253540 4789 scope.go:122] "RemoveContainer" containerID="6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996" Oct 30 19:24:02.255086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.255058 4789 generic.go:356] "Generic (PLEG): container finished" podID="60e3a351-1caf-459e-93e3-eed2414b6e31" containerID="d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd" exitCode=0 Oct 30 19:24:02.255181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.255094 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" event={"ID":"60e3a351-1caf-459e-93e3-eed2414b6e31","Type":"ContainerDied","Data":"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd"} Oct 30 19:24:02.255181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.255160 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" event={"ID":"60e3a351-1caf-459e-93e3-eed2414b6e31","Type":"ContainerDied","Data":"0f4dc7d15d00532b82ba6672fb8987d00d1aa425394884686a963fda812b7b06"} Oct 30 19:24:02.255320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.255106 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct" Oct 30 19:24:02.264448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.264356 4789 scope.go:122] "RemoveContainer" containerID="6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996" Oct 30 19:24:02.264697 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:02.264669 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996\": container with ID starting with 6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996 not found: ID does not exist" containerID="6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996" Oct 30 19:24:02.264797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.264701 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996"} err="failed to get container status \"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996\": rpc error: code = NotFound desc = could not find container \"6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996\": container with ID starting with 6193f6e9c98f96460dfc395b65aaa41975863215fb84b5a54791d8f540e2e996 not found: ID does not exist" Oct 30 19:24:02.264797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.264719 4789 scope.go:122] "RemoveContainer" containerID="d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd" Oct 30 19:24:02.276769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.276752 4789 scope.go:122] "RemoveContainer" containerID="d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd" Oct 30 19:24:02.277050 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:02.277031 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd\": container with ID starting with d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd not found: ID does not exist" containerID="d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd" Oct 30 19:24:02.277101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.277058 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd"} err="failed to get container status \"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd\": rpc error: code = NotFound desc = could not find container \"d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd\": container with ID starting with d95e0d3eccb0b0e1636c6c70b9e016fa5eaab68ad576dd32c59a44a1ccee42bd not found: ID does not exist" Oct 30 19:24:02.408661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.408632 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:24:02.455025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.454989 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-539084-5d86c7b68-m7zct"] Oct 30 19:24:02.561747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.561664 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:24:02.565627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:02.565603 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-56c8c8f44c-4klvb"] Oct 30 19:24:03.669385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:03.669355 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" path="/var/lib/kubelet/pods/12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb/volumes" Oct 30 19:24:03.669761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:03.669731 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e3a351-1caf-459e-93e3-eed2414b6e31" path="/var/lib/kubelet/pods/60e3a351-1caf-459e-93e3-eed2414b6e31/volumes" Oct 30 19:24:05.206771 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.206735 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:24:05.306925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.306891 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:24:05.841043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841012 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:24:05.841623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841604 4789 cpu_manager.go:414] "containerMap: removing container" podUID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" containerName="payload-pre-processing" Oct 30 19:24:05.841623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841622 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" containerName="payload-pre-processing" Oct 30 19:24:05.841762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841644 4789 cpu_manager.go:414] "containerMap: removing container" podUID="88dceeb4-1e61-43ed-9fe8-d76921458c82" containerName="revoke-keys" Oct 30 19:24:05.841762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841651 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88dceeb4-1e61-43ed-9fe8-d76921458c82" containerName="revoke-keys" Oct 30 19:24:05.841762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841668 4789 cpu_manager.go:414] "containerMap: removing container" podUID="60e3a351-1caf-459e-93e3-eed2414b6e31" containerName="payload-processing" Oct 30 19:24:05.841762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841675 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e3a351-1caf-459e-93e3-eed2414b6e31" containerName="payload-processing" Oct 30 19:24:05.841890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841768 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f1f6a2-3248-4d3b-bb35-5f8db8a09eeb" containerName="payload-pre-processing" Oct 30 19:24:05.841890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841778 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="88dceeb4-1e61-43ed-9fe8-d76921458c82" containerName="revoke-keys" Oct 30 19:24:05.841890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.841787 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e3a351-1caf-459e-93e3-eed2414b6e31" containerName="payload-processing" Oct 30 19:24:05.846643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.846627 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:05.860097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.860075 4789 scope.go:122] "RemoveContainer" containerID="417744d0e1a7ed504816bdb6ee080743aeedbfed978fcd5294c11c8b25e0f22a" Oct 30 19:24:05.870175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.870153 4789 scope.go:122] "RemoveContainer" containerID="8a834b10f6b4257868a12df6f53f9631b9570b42c46939c0a94642a48c5e133a" Oct 30 19:24:05.878653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.878629 4789 scope.go:122] "RemoveContainer" containerID="c9fc5aeda50c0edb66c9e768bfa51b9274ae8d73baf552313536688a8f7c852a" Oct 30 19:24:05.898378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.898353 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-65d741-dockercfg-srq42\"" Oct 30 19:24:05.898525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.898468 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-65d741\"" Oct 30 19:24:05.902202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.902175 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:24:05.992838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.992801 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:05.993035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:05.993014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkf6\" (UniqueName: \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.094692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.094607 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkf6\" (UniqueName: \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.094845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.094697 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.095453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.095427 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.152925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.152898 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkf6\" (UniqueName: \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\") pod \"payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.161802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.161780 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:06.539865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:06.539831 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:24:06.546543 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:24:06.546516 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df6681d_f075_4417_8b08_e2d169d9b16a.slice/crio-723719fb3a81396a4174d38c076d14f80da6a69ea4e264ee77ad020669557f23 WatchSource:0}: Error finding container 723719fb3a81396a4174d38c076d14f80da6a69ea4e264ee77ad020669557f23: Status 404 returned error can't find the container with id 723719fb3a81396a4174d38c076d14f80da6a69ea4e264ee77ad020669557f23 Oct 30 19:24:07.298186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:07.298135 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" event={"ID":"3df6681d-f075-4417-8b08-e2d169d9b16a","Type":"ContainerStarted","Data":"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f"} Oct 30 19:24:07.298406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:07.298200 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" event={"ID":"3df6681d-f075-4417-8b08-e2d169d9b16a","Type":"ContainerStarted","Data":"723719fb3a81396a4174d38c076d14f80da6a69ea4e264ee77ad020669557f23"} Oct 30 19:24:07.298406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:07.298258 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:07.408361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:07.408311 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" podStartSLOduration=2.408293692 podStartE2EDuration="2.408293692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:07.407706796 +0000 UTC m=+2283.909617281" watchObservedRunningTime="2026-10-30 19:24:07.408293692 +0000 UTC m=+2283.910204178" Oct 30 19:24:10.208036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.207993 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" podUID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" containerName="payload-processing" containerID="cri-o://e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848" gracePeriod=25 Oct 30 19:24:10.307626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.307566 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" podUID="8c45075f-87ac-46b6-9740-15d889971f7a" containerName="payload-pre-processing" containerID="cri-o://764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813" gracePeriod=25 Oct 30 19:24:10.485708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.485677 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:24:10.614419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.614391 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:24:10.656737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.656711 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\") pod \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " Oct 30 19:24:10.656864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.656757 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\" (UniqueName: \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\") pod \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\" (UID: \"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d\") " Oct 30 19:24:10.657068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.657047 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume" pod "aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" (UID: "aaf6a1fd-fc09-42fd-afe4-635efdfcd52d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:10.657353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.657335 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:10.658913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.658894 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k" pod "aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" (UID: "aaf6a1fd-fc09-42fd-afe4-635efdfcd52d"). InnerVolumeSpecName "kube-api-access-xln2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:10.758463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.758385 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\") pod \"8c45075f-87ac-46b6-9740-15d889971f7a\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " Oct 30 19:24:10.758463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.758450 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\" (UniqueName: \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\") pod \"8c45075f-87ac-46b6-9740-15d889971f7a\" (UID: \"8c45075f-87ac-46b6-9740-15d889971f7a\") " Oct 30 19:24:10.758763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.758748 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xln2k\" (UniqueName: \"kubernetes.io/projected/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d-kube-api-access-xln2k\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:10.758853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.758811 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume" pod "8c45075f-87ac-46b6-9740-15d889971f7a" (UID: "8c45075f-87ac-46b6-9740-15d889971f7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:10.760587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.760557 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf" pod "8c45075f-87ac-46b6-9740-15d889971f7a" (UID: "8c45075f-87ac-46b6-9740-15d889971f7a"). InnerVolumeSpecName "kube-api-access-8nlnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:10.860377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.860350 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c45075f-87ac-46b6-9740-15d889971f7a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:10.860377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:10.860377 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlnf\" (UniqueName: \"kubernetes.io/projected/8c45075f-87ac-46b6-9740-15d889971f7a-kube-api-access-8nlnf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:11.326550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.326521 4789 generic.go:356] "Generic (PLEG): container finished" podID="8c45075f-87ac-46b6-9740-15d889971f7a" containerID="764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813" exitCode=0 Oct 30 19:24:11.326942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.326561 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" event={"ID":"8c45075f-87ac-46b6-9740-15d889971f7a","Type":"ContainerDied","Data":"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813"} Oct 30 19:24:11.326942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.326617 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" event={"ID":"8c45075f-87ac-46b6-9740-15d889971f7a","Type":"ContainerDied","Data":"e20149c9a92e0d9f8cd729174060d68ccec4436ff18456a4462579819ab0e3fc"} Oct 30 19:24:11.326942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.326646 4789 scope.go:122] "RemoveContainer" containerID="764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813" Oct 30 19:24:11.326942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.326567 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj" Oct 30 19:24:11.328182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.328162 4789 generic.go:356] "Generic (PLEG): container finished" podID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" containerID="e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848" exitCode=0 Oct 30 19:24:11.328319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.328197 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" Oct 30 19:24:11.328319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.328249 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" event={"ID":"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d","Type":"ContainerDied","Data":"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848"} Oct 30 19:24:11.328319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.328294 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f" event={"ID":"aaf6a1fd-fc09-42fd-afe4-635efdfcd52d","Type":"ContainerDied","Data":"70a9817927dea22de22cdad5df988202bf4b6501912f2bba8a36130a288a4235"} Oct 30 19:24:11.336386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.336368 4789 scope.go:122] "RemoveContainer" containerID="764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813" Oct 30 19:24:11.336664 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:11.336645 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813\": container with ID starting with 764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813 not found: ID does not exist" containerID="764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813" Oct 30 19:24:11.336740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.336675 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813"} err="failed to get container status \"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813\": rpc error: code = NotFound desc = could not find container \"764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813\": container with ID starting with 764b5bd1c9827d363eaab2b3889f63849fed561aa73bd5129fce54d03c728813 not found: ID does not exist" Oct 30 19:24:11.336740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.336697 4789 scope.go:122] "RemoveContainer" containerID="e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848" Oct 30 19:24:11.345393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.345375 4789 scope.go:122] "RemoveContainer" containerID="e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848" Oct 30 19:24:11.345651 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:11.345633 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848\": container with ID starting with e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848 not found: ID does not exist" containerID="e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848" Oct 30 19:24:11.345716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.345658 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848"} err="failed to get container status \"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848\": rpc error: code = NotFound desc = could not find container \"e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848\": container with ID starting with e58e58ab5d7a8b06d275acd24e240727fc41ded9328f385cdb88e7e2a1a93848 not found: ID does not exist" Oct 30 19:24:11.481737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.481656 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:24:11.528289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.528258 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-496c5b-6f55cbfc44-vzf6f"] Oct 30 19:24:11.589020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.588986 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:24:11.597792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.597759 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-7599496c4c-6cqpj"] Oct 30 19:24:11.668581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.668550 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c45075f-87ac-46b6-9740-15d889971f7a" path="/var/lib/kubelet/pods/8c45075f-87ac-46b6-9740-15d889971f7a/volumes" Oct 30 19:24:11.668988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:11.668967 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" path="/var/lib/kubelet/pods/aaf6a1fd-fc09-42fd-afe4-635efdfcd52d/volumes" Oct 30 19:24:18.196716 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:18.196639 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:24:18.305602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:18.305579 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:24:23.034224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:23.034185 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:24:23.034712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:23.034470 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" containerID="cri-o://8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623" gracePeriod=30 Oct 30 19:24:24.338523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:24.338491 4789 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-54987 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:24:24.338900 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:24.338566 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:24:28.294757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.294729 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:24:28.439659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.439628 4789 generic.go:356] "Generic (PLEG): container finished" podID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerID="8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623" exitCode=0 Oct 30 19:24:28.439812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.439755 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" event={"ID":"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c","Type":"ContainerDied","Data":"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623"} Oct 30 19:24:28.439812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.439803 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" event={"ID":"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c","Type":"ContainerDied","Data":"201fefe78efc9007ba1c168bfd4679599427253568b9e935b268e62b960ee3bb"} Oct 30 19:24:28.439897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.439885 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987" Oct 30 19:24:28.440023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.440011 4789 scope.go:122] "RemoveContainer" containerID="8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623" Oct 30 19:24:28.451536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451517 4789 scope.go:122] "RemoveContainer" containerID="8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623" Oct 30 19:24:28.451629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451514 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.451702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451636 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.451702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451659 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.451702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451692 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.451864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451746 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.451864 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:28.451787 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623\": container with ID starting with 8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623 not found: ID does not exist" containerID="8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623" Oct 30 19:24:28.451864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451816 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623"} err="failed to get container status \"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623\": rpc error: code = NotFound desc = could not find container \"8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623\": container with ID starting with 8080a64951d39329ebd4b6b89ffcdfd290b4bfece82983e526b28d42dca81623 not found: ID does not exist" Oct 30 19:24:28.451864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451792 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.452089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451882 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.452089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.451995 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:28.452360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452315 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:28.452360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452331 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:28.452360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452342 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:28.452360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452355 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.452621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452387 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\") pod \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\" (UID: \"0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c\") " Oct 30 19:24:28.452621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452449 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:28.452853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452836 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.452914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452859 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.452914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452873 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.452914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452888 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.452914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.452902 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.454338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.454315 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "kube-api-access-n6p8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:28.454928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.454909 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:24:28.455020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.455011 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:28.455537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.455520 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token" pod "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" (UID: "0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:28.554445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.554418 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.554445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.554443 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6p8x\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-kube-api-access-n6p8x\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.554595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.554454 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.554595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.554463 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:28.856518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.856484 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:24:28.904155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:28.904127 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-54987"] Oct 30 19:24:29.669045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:29.669010 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" path="/var/lib/kubelet/pods/0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c/volumes" Oct 30 19:24:36.878122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878086 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:24:36.878594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878581 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" Oct 30 19:24:36.878594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878593 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" Oct 30 19:24:36.878670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878605 4789 cpu_manager.go:414] "containerMap: removing container" podUID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" containerName="payload-processing" Oct 30 19:24:36.878670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878610 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" containerName="payload-processing" Oct 30 19:24:36.878670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878621 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8c45075f-87ac-46b6-9740-15d889971f7a" containerName="payload-pre-processing" Oct 30 19:24:36.878670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878626 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c45075f-87ac-46b6-9740-15d889971f7a" containerName="payload-pre-processing" Oct 30 19:24:36.878845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878712 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec37f1c-c7d6-4e7f-bed0-7ace8b136e7c" containerName="istio-proxy" Oct 30 19:24:36.878845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878721 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf6a1fd-fc09-42fd-afe4-635efdfcd52d" containerName="payload-processing" Oct 30 19:24:36.878845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.878727 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c45075f-87ac-46b6-9740-15d889971f7a" containerName="payload-pre-processing" Oct 30 19:24:36.883750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.883729 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:36.981065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.981040 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7a790b\"" Oct 30 19:24:36.983786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:36.983764 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:24:37.042702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.042675 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.042835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.042803 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcmn\" (UniqueName: \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.143488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.143430 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcmn\" (UniqueName: \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.143611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.143541 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.144150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.144133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.200935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.200912 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcmn\" (UniqueName: \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\") pod \"payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.369919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.369890 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:24:37.373937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.373918 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.426478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.426423 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7a790b-dockercfg-6ff2q\"" Oct 30 19:24:37.431140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.431122 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:24:37.497955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.497922 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:37.547733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.547701 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.547875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.547748 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdlf\" (UniqueName: \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.649073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.649044 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.649245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.649080 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdlf\" (UniqueName: \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.649772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.649752 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.706897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.706872 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdlf\" (UniqueName: \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\") pod \"payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:37.880819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.880794 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:24:37.889408 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:24:37.889379 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb356a8_a99a_4b18_b5c1_47212269ea45.slice/crio-1fdb889b664a7e7509def2b69efa7f390cecda8bf3a205a6f1a82107b1b31c05 WatchSource:0}: Error finding container 1fdb889b664a7e7509def2b69efa7f390cecda8bf3a205a6f1a82107b1b31c05: Status 404 returned error can't find the container with id 1fdb889b664a7e7509def2b69efa7f390cecda8bf3a205a6f1a82107b1b31c05 Oct 30 19:24:37.988275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:37.988220 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:38.367119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.367031 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:24:38.373886 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:24:38.373847 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884841f5_5632_4f49_a964_96c7f5e6ae0e.slice/crio-730612461ceb0e290d3f306574bbb71080f937bedd0baf3407ca98d4b7bc6af8 WatchSource:0}: Error finding container 730612461ceb0e290d3f306574bbb71080f937bedd0baf3407ca98d4b7bc6af8: Status 404 returned error can't find the container with id 730612461ceb0e290d3f306574bbb71080f937bedd0baf3407ca98d4b7bc6af8 Oct 30 19:24:38.509446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.509411 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" event={"ID":"4eb356a8-a99a-4b18-b5c1-47212269ea45","Type":"ContainerStarted","Data":"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915"} Oct 30 19:24:38.509629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.509474 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:38.509629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.509497 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" event={"ID":"4eb356a8-a99a-4b18-b5c1-47212269ea45","Type":"ContainerStarted","Data":"1fdb889b664a7e7509def2b69efa7f390cecda8bf3a205a6f1a82107b1b31c05"} Oct 30 19:24:38.511607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.511577 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" event={"ID":"884841f5-5632-4f49-a964-96c7f5e6ae0e","Type":"ContainerStarted","Data":"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b"} Oct 30 19:24:38.511768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.511628 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" event={"ID":"884841f5-5632-4f49-a964-96c7f5e6ae0e","Type":"ContainerStarted","Data":"730612461ceb0e290d3f306574bbb71080f937bedd0baf3407ca98d4b7bc6af8"} Oct 30 19:24:38.511768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.511672 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:38.619092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.619020 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h"] Oct 30 19:24:38.623451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.623423 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.631779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.631650 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" podStartSLOduration=2.631634427 podStartE2EDuration="2.631634427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:38.627908789 +0000 UTC m=+2315.129819275" watchObservedRunningTime="2026-10-30 19:24:38.631634427 +0000 UTC m=+2315.133544914" Oct 30 19:24:38.673568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.673531 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:24:38.673708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.673565 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:24:38.673708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.673598 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:24:38.673903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.673882 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:24:38.676995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.676971 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h"] Oct 30 19:24:38.762156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.762112 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.762367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.762283 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6fr\" (UniqueName: \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.762466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.762405 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.846755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.846705 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" podStartSLOduration=1.846691258 podStartE2EDuration="1.846691258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:38.845921812 +0000 UTC m=+2315.347832299" watchObservedRunningTime="2026-10-30 19:24:38.846691258 +0000 UTC m=+2315.348601741" Oct 30 19:24:38.863614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.863588 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6fr\" (UniqueName: \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.863772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.863639 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.863772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.863738 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.864320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.864291 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.866189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.866166 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.921628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.921564 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6fr\" (UniqueName: \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\") pod \"maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:38.937787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:38.937760 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:39.327877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:39.327844 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h"] Oct 30 19:24:39.333888 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:24:39.333864 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ccd31_93d8_4834_bb77_d40b39b79696.slice/crio-071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5 WatchSource:0}: Error finding container 071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5: Status 404 returned error can't find the container with id 071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5 Oct 30 19:24:39.520738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:39.520702 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" event={"ID":"011ccd31-93d8-4834-bb77-d40b39b79696","Type":"ContainerStarted","Data":"071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5"} Oct 30 19:24:40.529118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:40.529086 4789 generic.go:356] "Generic (PLEG): container finished" podID="011ccd31-93d8-4834-bb77-d40b39b79696" containerID="65dc37337101f73a4549a7c7eda72b5b2495c1f0f70118251b634af3eb56b4f4" exitCode=0 Oct 30 19:24:40.529519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:40.529132 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" event={"ID":"011ccd31-93d8-4834-bb77-d40b39b79696","Type":"ContainerDied","Data":"65dc37337101f73a4549a7c7eda72b5b2495c1f0f70118251b634af3eb56b4f4"} Oct 30 19:24:41.693977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.693947 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:41.702322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.702301 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\") pod \"011ccd31-93d8-4834-bb77-d40b39b79696\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " Oct 30 19:24:41.702450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.702358 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\" (UniqueName: \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\") pod \"011ccd31-93d8-4834-bb77-d40b39b79696\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " Oct 30 19:24:41.702522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.702506 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\") pod \"011ccd31-93d8-4834-bb77-d40b39b79696\" (UID: \"011ccd31-93d8-4834-bb77-d40b39b79696\") " Oct 30 19:24:41.702898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.702867 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca" pod "011ccd31-93d8-4834-bb77-d40b39b79696" (UID: "011ccd31-93d8-4834-bb77-d40b39b79696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:41.704338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.704313 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca" pod "011ccd31-93d8-4834-bb77-d40b39b79696" (UID: "011ccd31-93d8-4834-bb77-d40b39b79696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:24:41.704453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.704433 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr" pod "011ccd31-93d8-4834-bb77-d40b39b79696" (UID: "011ccd31-93d8-4834-bb77-d40b39b79696"). InnerVolumeSpecName "kube-api-access-pl6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:41.804147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.804082 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:41.804147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.804107 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6fr\" (UniqueName: \"kubernetes.io/projected/011ccd31-93d8-4834-bb77-d40b39b79696-kube-api-access-pl6fr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:41.804147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:41.804117 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011ccd31-93d8-4834-bb77-d40b39b79696-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:42.544413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:42.544387 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" Oct 30 19:24:42.544582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:42.544384 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h" event={"ID":"011ccd31-93d8-4834-bb77-d40b39b79696","Type":"ContainerDied","Data":"071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5"} Oct 30 19:24:42.544582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:42.544509 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071e3c02342bfb5f2a90db1cc142471d87882fdfc1e508f74e7f2ca5c2acfca5" Oct 30 19:24:48.196100 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:48.196063 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:24:49.522929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:49.522898 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:24:49.523320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:49.523068 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:24:50.216984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:50.216950 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:24:55.217821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.217773 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" podUID="8105bd64-c702-4a7e-959b-2de393c39690" containerName="payload-pre-processing" containerID="cri-o://f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4" gracePeriod=25 Oct 30 19:24:55.488511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.488488 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:24:55.548883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.548858 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\" (UniqueName: \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\") pod \"8105bd64-c702-4a7e-959b-2de393c39690\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " Oct 30 19:24:55.548994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.548959 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\") pod \"8105bd64-c702-4a7e-959b-2de393c39690\" (UID: \"8105bd64-c702-4a7e-959b-2de393c39690\") " Oct 30 19:24:55.549334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.549310 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume" pod "8105bd64-c702-4a7e-959b-2de393c39690" (UID: "8105bd64-c702-4a7e-959b-2de393c39690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:55.550832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.550809 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw" pod "8105bd64-c702-4a7e-959b-2de393c39690" (UID: "8105bd64-c702-4a7e-959b-2de393c39690"). InnerVolumeSpecName "kube-api-access-9wvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:55.623816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.623779 4789 generic.go:356] "Generic (PLEG): container finished" podID="8105bd64-c702-4a7e-959b-2de393c39690" containerID="f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4" exitCode=0 Oct 30 19:24:55.624008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.623837 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" event={"ID":"8105bd64-c702-4a7e-959b-2de393c39690","Type":"ContainerDied","Data":"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4"} Oct 30 19:24:55.624008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.623887 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" Oct 30 19:24:55.624008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.623916 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l" event={"ID":"8105bd64-c702-4a7e-959b-2de393c39690","Type":"ContainerDied","Data":"b2be9aebb2fc750709bc4d2130f493e61c5a7d39873e7129ae910f8607f77bb0"} Oct 30 19:24:55.624008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.623945 4789 scope.go:122] "RemoveContainer" containerID="f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4" Oct 30 19:24:55.640435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.640373 4789 scope.go:122] "RemoveContainer" containerID="f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4" Oct 30 19:24:55.640796 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:24:55.640729 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4\": container with ID starting with f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4 not found: ID does not exist" containerID="f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4" Oct 30 19:24:55.640796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.640768 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4"} err="failed to get container status \"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4\": rpc error: code = NotFound desc = could not find container \"f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4\": container with ID starting with f0d01dfa1db9f07bdb1fded1dd6f82dd8a9126546061ddda84c007678c94a4a4 not found: ID does not exist" Oct 30 19:24:55.650509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.650489 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8105bd64-c702-4a7e-959b-2de393c39690-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:55.650594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.650510 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvnw\" (UniqueName: \"kubernetes.io/projected/8105bd64-c702-4a7e-959b-2de393c39690-kube-api-access-9wvnw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:55.782256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.782170 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:24:55.789179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:55.789160 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-324ebe-5c6d944667-wfv5l"] Oct 30 19:24:57.668008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:24:57.667979 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8105bd64-c702-4a7e-959b-2de393c39690" path="/var/lib/kubelet/pods/8105bd64-c702-4a7e-959b-2de393c39690/volumes" Oct 30 19:25:02.777834 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.777803 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:25:02.778459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778440 4789 cpu_manager.go:414] "containerMap: removing container" podUID="011ccd31-93d8-4834-bb77-d40b39b79696" containerName="revoke-keys" Oct 30 19:25:02.778459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778458 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011ccd31-93d8-4834-bb77-d40b39b79696" containerName="revoke-keys" Oct 30 19:25:02.778600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778474 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8105bd64-c702-4a7e-959b-2de393c39690" containerName="payload-pre-processing" Oct 30 19:25:02.778600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778480 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8105bd64-c702-4a7e-959b-2de393c39690" containerName="payload-pre-processing" Oct 30 19:25:02.778676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778600 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="011ccd31-93d8-4834-bb77-d40b39b79696" containerName="revoke-keys" Oct 30 19:25:02.778676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.778612 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8105bd64-c702-4a7e-959b-2de393c39690" containerName="payload-pre-processing" Oct 30 19:25:02.782178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.782161 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.838228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.834191 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2a5c13-openshift-default-dockercfg-cdlb8\"" Oct 30 19:25:02.889774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.889740 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:25:02.932273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.932012 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbb2s\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.932273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.932089 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940570 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940629 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940710 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940760 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940870 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940905 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:02.948234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:02.940990 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.041937 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042011 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbb2s\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042062 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042088 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042119 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042190 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042251 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042350 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042387 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.042871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.042844 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.043122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.043095 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.043378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.043352 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.043544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.043523 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.044391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.044371 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.046535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.046512 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.047070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.047048 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.101157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.101123 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbb2s\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.101369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.101345 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\") pod \"e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.395570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.395489 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:03.833732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.833696 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:25:03.839168 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:03.839129 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3882b5_54e1_47ed_8274_9d1f0618f473.slice/crio-bfc72102a62cd10047f018b133557a8fbd51287df6b007dc6e87f59b6eb88e5b WatchSource:0}: Error finding container bfc72102a62cd10047f018b133557a8fbd51287df6b007dc6e87f59b6eb88e5b: Status 404 returned error can't find the container with id bfc72102a62cd10047f018b133557a8fbd51287df6b007dc6e87f59b6eb88e5b Oct 30 19:25:03.842320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.842285 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:03.842405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.842354 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:03.842405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:03.842401 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:25:04.689618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.689579 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerStarted","Data":"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa"} Oct 30 19:25:04.689790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.689632 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerStarted","Data":"bfc72102a62cd10047f018b133557a8fbd51287df6b007dc6e87f59b6eb88e5b"} Oct 30 19:25:04.702697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.702663 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:25:04.706643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.706625 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:04.785072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.785050 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-539084\"" Oct 30 19:25:04.785348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.785331 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-539084-dockercfg-kb9h8\"" Oct 30 19:25:04.788629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.788604 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:25:04.833718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.833677 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podStartSLOduration=2.8336673 podStartE2EDuration="2.8336673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:04.811313028 +0000 UTC m=+2341.313223514" watchObservedRunningTime="2026-10-30 19:25:04.8336673 +0000 UTC m=+2341.335577844" Oct 30 19:25:04.869165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.869142 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfc22\" (UniqueName: \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:04.869487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.869298 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:04.970693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.970608 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:04.970693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.970667 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfc22\" (UniqueName: \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:04.971485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:04.971454 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:05.028858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.028830 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfc22\" (UniqueName: \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\") pod \"payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:05.320795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.320721 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:05.396438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.396411 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:05.402226 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.402189 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:05.698511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.698481 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:05.699961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.699715 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:25:05.706244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.701584 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:25:05.708800 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:05.708773 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606895ee_8976_4648_b97d_8ce9d917efad.slice/crio-e015d21653fdc0e1e51217bd523a6cd09602922ea4912cb3e979adb8ea8ce813 WatchSource:0}: Error finding container e015d21653fdc0e1e51217bd523a6cd09602922ea4912cb3e979adb8ea8ce813: Status 404 returned error can't find the container with id e015d21653fdc0e1e51217bd523a6cd09602922ea4912cb3e979adb8ea8ce813 Oct 30 19:25:05.931584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.931558 4789 scope.go:122] "RemoveContainer" containerID="261b1c118c7060f49d31cfa37a0de8591e6be6f43a7a98eb2a4eb1acb2969f38" Oct 30 19:25:05.940851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:05.940833 4789 scope.go:122] "RemoveContainer" containerID="1e9fdac30ecd61b9e6f4f435b639c64ddac3dd1a53787a6ad7b0279c6d73f33c" Oct 30 19:25:06.706147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:06.706112 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" event={"ID":"606895ee-8976-4648-b97d-8ce9d917efad","Type":"ContainerStarted","Data":"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d"} Oct 30 19:25:06.706344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:06.706158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" event={"ID":"606895ee-8976-4648-b97d-8ce9d917efad","Type":"ContainerStarted","Data":"e015d21653fdc0e1e51217bd523a6cd09602922ea4912cb3e979adb8ea8ce813"} Oct 30 19:25:06.706344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:06.706246 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:06.816283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:06.816235 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" podStartSLOduration=2.81619607 podStartE2EDuration="2.81619607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:06.814996532 +0000 UTC m=+2343.316907019" watchObservedRunningTime="2026-10-30 19:25:06.81619607 +0000 UTC m=+2343.318106592" Oct 30 19:25:11.463536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.463504 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:25:11.470929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.470908 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.526279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.526243 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:25:11.568259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.568231 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2a5c13\"" Oct 30 19:25:11.568498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.568478 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2a5c13-dockercfg-m9q27\"" Oct 30 19:25:11.641355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.641322 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.641459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.641383 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7pq\" (UniqueName: \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.742447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.742386 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.742447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.742444 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7pq\" (UniqueName: \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.743024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.743004 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:11.800559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:11.800532 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7pq\" (UniqueName: \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\") pod \"payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:12.085600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.085523 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:12.469928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.469893 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:25:12.476703 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:12.476674 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b7f3bb_4b97_40e9_a6fb_30e813cd9344.slice/crio-dde28101b00d1710642393ae284cc290d73a966aa7f99573c53fb0a9d0e85da8 WatchSource:0}: Error finding container dde28101b00d1710642393ae284cc290d73a966aa7f99573c53fb0a9d0e85da8: Status 404 returned error can't find the container with id dde28101b00d1710642393ae284cc290d73a966aa7f99573c53fb0a9d0e85da8 Oct 30 19:25:12.752350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.752249 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" event={"ID":"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344","Type":"ContainerStarted","Data":"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b"} Oct 30 19:25:12.752350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.752314 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" event={"ID":"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344","Type":"ContainerStarted","Data":"dde28101b00d1710642393ae284cc290d73a966aa7f99573c53fb0a9d0e85da8"} Oct 30 19:25:12.752350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.752342 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:12.862749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:12.862703 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" podStartSLOduration=1.862690299 podStartE2EDuration="1.862690299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:12.86227207 +0000 UTC m=+2349.364182555" watchObservedRunningTime="2026-10-30 19:25:12.862690299 +0000 UTC m=+2349.364600785" Oct 30 19:25:16.290036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.289929 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:25:16.297697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.297251 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.351979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.351948 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:25:16.394429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.394398 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-496c5b\"" Oct 30 19:25:16.394588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.394549 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-496c5b-dockercfg-s9wss\"" Oct 30 19:25:16.398186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.398160 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrfw\" (UniqueName: \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.398392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.398373 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.499319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.499285 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrfw\" (UniqueName: \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.499490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.499461 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.500159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.500137 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.557646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.557587 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrfw\" (UniqueName: \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\") pod \"payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.618527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.618503 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:16.997067 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:16.997037 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:25:17.003545 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:17.003515 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea13fb92_b3c3_4857_8401_4031048352e3.slice/crio-028c71dd0d55a76337581b3cdefecf2090be86e2e50807d5da88a60d9a0f315c WatchSource:0}: Error finding container 028c71dd0d55a76337581b3cdefecf2090be86e2e50807d5da88a60d9a0f315c: Status 404 returned error can't find the container with id 028c71dd0d55a76337581b3cdefecf2090be86e2e50807d5da88a60d9a0f315c Oct 30 19:25:17.487710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.487669 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:25:17.488184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.487922 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" podUID="3df6681d-f075-4417-8b08-e2d169d9b16a" containerName="payload-pre-processing" containerID="cri-o://bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f" gracePeriod=30 Oct 30 19:25:17.713289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.713261 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:25:17.755097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.755040 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:25:17.783849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.783824 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" event={"ID":"ea13fb92-b3c3-4857-8401-4031048352e3","Type":"ContainerStarted","Data":"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de"} Oct 30 19:25:17.783981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.783884 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:17.783981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.783911 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" event={"ID":"ea13fb92-b3c3-4857-8401-4031048352e3","Type":"ContainerStarted","Data":"028c71dd0d55a76337581b3cdefecf2090be86e2e50807d5da88a60d9a0f315c"} Oct 30 19:25:17.785075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.785051 4789 generic.go:356] "Generic (PLEG): container finished" podID="3df6681d-f075-4417-8b08-e2d169d9b16a" containerID="bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f" exitCode=0 Oct 30 19:25:17.785167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.785092 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" Oct 30 19:25:17.785167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.785130 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" event={"ID":"3df6681d-f075-4417-8b08-e2d169d9b16a","Type":"ContainerDied","Data":"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f"} Oct 30 19:25:17.785324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.785173 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs" event={"ID":"3df6681d-f075-4417-8b08-e2d169d9b16a","Type":"ContainerDied","Data":"723719fb3a81396a4174d38c076d14f80da6a69ea4e264ee77ad020669557f23"} Oct 30 19:25:17.785324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.785195 4789 scope.go:122] "RemoveContainer" containerID="bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f" Oct 30 19:25:17.794543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.794521 4789 scope.go:122] "RemoveContainer" containerID="bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f" Oct 30 19:25:17.794864 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:25:17.794829 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f\": container with ID starting with bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f not found: ID does not exist" containerID="bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f" Oct 30 19:25:17.794950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.794874 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f"} err="failed to get container status \"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f\": rpc error: code = NotFound desc = could not find container \"bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f\": container with ID starting with bfe2cd1e49764016f615d9a5ef4ab110ab33ed3da6ed4887c02b67e7c2881a6f not found: ID does not exist" Oct 30 19:25:17.814931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.814910 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\" (UniqueName: \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\") pod \"3df6681d-f075-4417-8b08-e2d169d9b16a\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " Oct 30 19:25:17.815049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.815007 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\") pod \"3df6681d-f075-4417-8b08-e2d169d9b16a\" (UID: \"3df6681d-f075-4417-8b08-e2d169d9b16a\") " Oct 30 19:25:17.815394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.815371 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume" pod "3df6681d-f075-4417-8b08-e2d169d9b16a" (UID: "3df6681d-f075-4417-8b08-e2d169d9b16a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:17.817415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.817395 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6" pod "3df6681d-f075-4417-8b08-e2d169d9b16a" (UID: "3df6681d-f075-4417-8b08-e2d169d9b16a"). InnerVolumeSpecName "kube-api-access-fjkf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:17.916560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.916523 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkf6\" (UniqueName: \"kubernetes.io/projected/3df6681d-f075-4417-8b08-e2d169d9b16a-kube-api-access-fjkf6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:17.916560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:17.916558 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3df6681d-f075-4417-8b08-e2d169d9b16a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:18.049309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:18.047516 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" podStartSLOduration=2.047479562 podStartE2EDuration="2.047479562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:18.046179983 +0000 UTC m=+2354.548090468" watchObservedRunningTime="2026-10-30 19:25:18.047479562 +0000 UTC m=+2354.549390047" Oct 30 19:25:18.206766 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:25:18.206725 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:25:18.217471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:18.217445 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:25:18.222416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:18.222391 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-65d741-8676c86cb7-qlcxs"] Oct 30 19:25:19.081918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:19.081881 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf"] Oct 30 19:25:19.127999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:19.127973 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-65d741-hvzzf"] Oct 30 19:25:19.668186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:19.668157 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df6681d-f075-4417-8b08-e2d169d9b16a" path="/var/lib/kubelet/pods/3df6681d-f075-4417-8b08-e2d169d9b16a/volumes" Oct 30 19:25:19.668590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:19.668575 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88dceeb4-1e61-43ed-9fe8-d76921458c82" path="/var/lib/kubelet/pods/88dceeb4-1e61-43ed-9fe8-d76921458c82/volumes" Oct 30 19:25:20.633344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.633311 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs"] Oct 30 19:25:20.635073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.635044 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3df6681d-f075-4417-8b08-e2d169d9b16a" containerName="payload-pre-processing" Oct 30 19:25:20.635251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.635236 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df6681d-f075-4417-8b08-e2d169d9b16a" containerName="payload-pre-processing" Oct 30 19:25:20.635675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.635656 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df6681d-f075-4417-8b08-e2d169d9b16a" containerName="payload-pre-processing" Oct 30 19:25:20.641726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.641706 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.690418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.690396 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs"] Oct 30 19:25:20.733281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.733249 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:25:20.733380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.733333 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:25:20.733465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.733436 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:25:20.736784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.736767 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:25:20.743327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.743309 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jvw\" (UniqueName: \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.743397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.743338 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.743397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.743371 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.845062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.845042 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jvw\" (UniqueName: \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.845166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.845068 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.845166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.845104 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.845785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.845765 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.847382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.847364 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.903306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.903251 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jvw\" (UniqueName: \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\") pod \"maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:20.954295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:20.954274 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:21.330028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:21.329995 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs"] Oct 30 19:25:21.336643 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:21.336620 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45403a4_3462_47b2_a317_fe0c5c355520.slice/crio-682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137 WatchSource:0}: Error finding container 682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137: Status 404 returned error can't find the container with id 682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137 Oct 30 19:25:21.810757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:21.810718 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" event={"ID":"f45403a4-3462-47b2-a317-fe0c5c355520","Type":"ContainerStarted","Data":"0eb11c147c646cb0169d7769b098f5332673604f850e7ce55ca9d80e1640b7cd"} Oct 30 19:25:21.811082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:21.810769 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" event={"ID":"f45403a4-3462-47b2-a317-fe0c5c355520","Type":"ContainerStarted","Data":"682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137"} Oct 30 19:25:21.929294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:21.929247 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" podStartSLOduration=1.9292325259999998 podStartE2EDuration="1.929232526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:21.926073131 +0000 UTC m=+2358.427983624" watchObservedRunningTime="2026-10-30 19:25:21.929232526 +0000 UTC m=+2358.431143010" Oct 30 19:25:22.818773 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:22.818740 4789 generic.go:356] "Generic (PLEG): container finished" podID="f45403a4-3462-47b2-a317-fe0c5c355520" containerID="0eb11c147c646cb0169d7769b098f5332673604f850e7ce55ca9d80e1640b7cd" exitCode=0 Oct 30 19:25:22.819205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:22.818789 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" event={"ID":"f45403a4-3462-47b2-a317-fe0c5c355520","Type":"ContainerDied","Data":"0eb11c147c646cb0169d7769b098f5332673604f850e7ce55ca9d80e1640b7cd"} Oct 30 19:25:23.761358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:23.761331 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:25:23.975280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:23.975255 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:24.078382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.078297 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\") pod \"f45403a4-3462-47b2-a317-fe0c5c355520\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " Oct 30 19:25:24.078382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.078362 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\") pod \"f45403a4-3462-47b2-a317-fe0c5c355520\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " Oct 30 19:25:24.078579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.078474 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\" (UniqueName: \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\") pod \"f45403a4-3462-47b2-a317-fe0c5c355520\" (UID: \"f45403a4-3462-47b2-a317-fe0c5c355520\") " Oct 30 19:25:24.078742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.078714 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca" pod "f45403a4-3462-47b2-a317-fe0c5c355520" (UID: "f45403a4-3462-47b2-a317-fe0c5c355520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:24.080427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.080402 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca" pod "f45403a4-3462-47b2-a317-fe0c5c355520" (UID: "f45403a4-3462-47b2-a317-fe0c5c355520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:24.080577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.080490 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw" pod "f45403a4-3462-47b2-a317-fe0c5c355520" (UID: "f45403a4-3462-47b2-a317-fe0c5c355520"). InnerVolumeSpecName "kube-api-access-g2jvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:24.179662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.179630 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jvw\" (UniqueName: \"kubernetes.io/projected/f45403a4-3462-47b2-a317-fe0c5c355520-kube-api-access-g2jvw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:24.179662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.179656 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:24.179662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.179666 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45403a4-3462-47b2-a317-fe0c5c355520-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:24.831546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.831509 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" event={"ID":"f45403a4-3462-47b2-a317-fe0c5c355520","Type":"ContainerDied","Data":"682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137"} Oct 30 19:25:24.831546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.831545 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs" Oct 30 19:25:24.831810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:24.831559 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682340e0b0934beb68cfe9c4eb3b65babfbbea13424a00e35815e64f51bc9137" Oct 30 19:25:28.795495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:28.795458 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:25:46.849127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.849054 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:25:46.849753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.849734 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f45403a4-3462-47b2-a317-fe0c5c355520" containerName="revoke-keys" Oct 30 19:25:46.849796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.849755 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45403a4-3462-47b2-a317-fe0c5c355520" containerName="revoke-keys" Oct 30 19:25:46.849861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.849852 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45403a4-3462-47b2-a317-fe0c5c355520" containerName="revoke-keys" Oct 30 19:25:46.854586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.854567 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:46.932619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.932580 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:46.932966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.932652 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbh5l\" (UniqueName: \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:46.949787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.949763 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4833f3\"" Oct 30 19:25:46.957804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.957782 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4833f3-dockercfg-sqpn9\"" Oct 30 19:25:46.963079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:46.963051 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:25:47.034451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.034413 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:47.034569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.034491 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbh5l\" (UniqueName: \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:47.035065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.035042 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:47.094813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.094788 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbh5l\" (UniqueName: \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\") pod \"payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:47.168085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.168027 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:47.548260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.548227 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:25:47.553867 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:47.553836 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode042745e_ef0d_4f87_9b6c_fffa8c3d81b3.slice/crio-e6270e44d2d17dd1af7f4820b483de890c722714a10803baac80ca0bf3d07537 WatchSource:0}: Error finding container e6270e44d2d17dd1af7f4820b483de890c722714a10803baac80ca0bf3d07537: Status 404 returned error can't find the container with id e6270e44d2d17dd1af7f4820b483de890c722714a10803baac80ca0bf3d07537 Oct 30 19:25:47.986187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.986158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" event={"ID":"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3","Type":"ContainerStarted","Data":"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51"} Oct 30 19:25:47.986571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.986233 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" event={"ID":"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3","Type":"ContainerStarted","Data":"e6270e44d2d17dd1af7f4820b483de890c722714a10803baac80ca0bf3d07537"} Oct 30 19:25:47.986571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:47.986258 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:48.100148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:48.100094 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" podStartSLOduration=2.100079819 podStartE2EDuration="2.100079819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:48.097117072 +0000 UTC m=+2384.599027557" watchObservedRunningTime="2026-10-30 19:25:48.100079819 +0000 UTC m=+2384.601990305" Oct 30 19:25:48.197684 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:25:48.197648 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:25:55.984267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:55.984229 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w"] Oct 30 19:25:55.988356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:55.988337 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.029110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.029085 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.029199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.029120 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.029274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.029235 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6z5\" (UniqueName: \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.085984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.085959 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:25:56.085984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.085959 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:25:56.085984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.085982 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:25:56.086173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.085986 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:25:56.089094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.089072 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w"] Oct 30 19:25:56.129747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.129722 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.129842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.129758 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.129842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.129810 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6z5\" (UniqueName: \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.130477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.130461 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.132002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.131984 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.187706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.187685 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6z5\" (UniqueName: \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.301290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.301241 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:56.676295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:56.676220 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w"] Oct 30 19:25:56.681793 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:25:56.681765 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531ca2bb_b43c_47d5_a92b_0359afcef00f.slice/crio-8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b WatchSource:0}: Error finding container 8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b: Status 404 returned error can't find the container with id 8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b Oct 30 19:25:57.041702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:57.041668 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" event={"ID":"531ca2bb-b43c-47d5-a92b-0359afcef00f","Type":"ContainerStarted","Data":"8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b"} Oct 30 19:25:58.049397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:58.049365 4789 generic.go:356] "Generic (PLEG): container finished" podID="531ca2bb-b43c-47d5-a92b-0359afcef00f" containerID="edfd022582243466b73cbaeb160dc3353e64b044cc3b1ed37c7ce95213495de8" exitCode=0 Oct 30 19:25:58.049944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:58.049411 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" event={"ID":"531ca2bb-b43c-47d5-a92b-0359afcef00f","Type":"ContainerDied","Data":"edfd022582243466b73cbaeb160dc3353e64b044cc3b1ed37c7ce95213495de8"} Oct 30 19:25:58.996132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:58.996104 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:25:59.266660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.266625 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:25:59.363243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.363177 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\") pod \"531ca2bb-b43c-47d5-a92b-0359afcef00f\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " Oct 30 19:25:59.363412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.363304 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\" (UniqueName: \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\") pod \"531ca2bb-b43c-47d5-a92b-0359afcef00f\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " Oct 30 19:25:59.363412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.363372 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\") pod \"531ca2bb-b43c-47d5-a92b-0359afcef00f\" (UID: \"531ca2bb-b43c-47d5-a92b-0359afcef00f\") " Oct 30 19:25:59.363775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.363745 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca" pod "531ca2bb-b43c-47d5-a92b-0359afcef00f" (UID: "531ca2bb-b43c-47d5-a92b-0359afcef00f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:59.363916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.363891 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:59.365269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.365244 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca" pod "531ca2bb-b43c-47d5-a92b-0359afcef00f" (UID: "531ca2bb-b43c-47d5-a92b-0359afcef00f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:59.365704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.365688 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5" pod "531ca2bb-b43c-47d5-a92b-0359afcef00f" (UID: "531ca2bb-b43c-47d5-a92b-0359afcef00f"). InnerVolumeSpecName "kube-api-access-tn6z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:59.465425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.465401 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/531ca2bb-b43c-47d5-a92b-0359afcef00f-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:59.465425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:25:59.465425 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6z5\" (UniqueName: \"kubernetes.io/projected/531ca2bb-b43c-47d5-a92b-0359afcef00f-kube-api-access-tn6z5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:00.065177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:00.065149 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" Oct 30 19:26:00.065355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:00.065183 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w" event={"ID":"531ca2bb-b43c-47d5-a92b-0359afcef00f","Type":"ContainerDied","Data":"8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b"} Oct 30 19:26:00.065355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:00.065251 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fdb7ce8fafb2a7bfb4f173e5871b7b3544f70faacbb7a2d37df400b5b49c27b" Oct 30 19:26:04.309926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:04.309898 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:26:04.325935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:04.325913 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:26:18.197987 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:26:18.197953 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:26:38.905596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:38.905556 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:26:38.905991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:38.905820 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" podUID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" containerName="payload-processing" containerID="cri-o://7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3" gracePeriod=30 Oct 30 19:26:39.225615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.225581 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:26:39.298415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.298377 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\" (UniqueName: \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\") pod \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " Oct 30 19:26:39.298605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.298526 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\") pod \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\" (UID: \"318bd54a-e1eb-4fd2-81ab-d767efe33c63\") " Oct 30 19:26:39.298895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.298865 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume" pod "318bd54a-e1eb-4fd2-81ab-d767efe33c63" (UID: "318bd54a-e1eb-4fd2-81ab-d767efe33c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:39.299017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.298954 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318bd54a-e1eb-4fd2-81ab-d767efe33c63-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:39.301025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.300995 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst" pod "318bd54a-e1eb-4fd2-81ab-d767efe33c63" (UID: "318bd54a-e1eb-4fd2-81ab-d767efe33c63"). InnerVolumeSpecName "kube-api-access-fnqst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:39.330817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.330788 4789 generic.go:356] "Generic (PLEG): container finished" podID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" containerID="7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3" exitCode=0 Oct 30 19:26:39.330965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.330841 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" Oct 30 19:26:39.330965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.330884 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" event={"ID":"318bd54a-e1eb-4fd2-81ab-d767efe33c63","Type":"ContainerDied","Data":"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3"} Oct 30 19:26:39.330965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.330944 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb" event={"ID":"318bd54a-e1eb-4fd2-81ab-d767efe33c63","Type":"ContainerDied","Data":"fa7b382b7e9a18a694120d7f5a589361f4dbe5f078c91d16399d9be74f0afbc4"} Oct 30 19:26:39.331116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.330976 4789 scope.go:122] "RemoveContainer" containerID="7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3" Oct 30 19:26:39.346120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.346093 4789 scope.go:122] "RemoveContainer" containerID="7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3" Oct 30 19:26:39.346485 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:26:39.346456 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3\": container with ID starting with 7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3 not found: ID does not exist" containerID="7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3" Oct 30 19:26:39.346585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.346502 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3"} err="failed to get container status \"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3\": rpc error: code = NotFound desc = could not find container \"7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3\": container with ID starting with 7f323e2faa9ee8a798b2007a1f6e52b440e7de1019ea82a23a90822cfe77a0e3 not found: ID does not exist" Oct 30 19:26:39.445670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.445579 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqst\" (UniqueName: \"kubernetes.io/projected/318bd54a-e1eb-4fd2-81ab-d767efe33c63-kube-api-access-fnqst\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:39.497666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.497631 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:26:39.500369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.500342 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-905566-74bd75f45b-9gtpb"] Oct 30 19:26:39.671956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:39.671923 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" path="/var/lib/kubelet/pods/318bd54a-e1eb-4fd2-81ab-d767efe33c63/volumes" Oct 30 19:26:40.632408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:40.632373 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs"] Oct 30 19:26:40.676762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:40.676720 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-905566-l7hgs"] Oct 30 19:26:41.348957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.348887 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerID="06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa" exitCode=0 Oct 30 19:26:41.349136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.348959 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerDied","Data":"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa"} Oct 30 19:26:41.349368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.349354 4789 scope.go:122] "RemoveContainer" containerID="06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa" Oct 30 19:26:41.350813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.350788 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:41.350878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.350854 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:41.350914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.350888 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:26:41.681667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.679311 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45403a4-3462-47b2-a317-fe0c5c355520" path="/var/lib/kubelet/pods/f45403a4-3462-47b2-a317-fe0c5c355520/volumes" Oct 30 19:26:41.804804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.804766 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h"] Oct 30 19:26:41.809038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:41.809007 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-324ebe-nmg8h"] Oct 30 19:26:42.357061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:42.357028 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerStarted","Data":"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7"} Oct 30 19:26:42.402234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:42.399632 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:26:42.402234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:42.399759 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:42.402234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:42.399800 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:43.406769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:43.406730 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:43.407400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:43.407368 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:43.667821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:43.667744 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011ccd31-93d8-4834-bb77-d40b39b79696" path="/var/lib/kubelet/pods/011ccd31-93d8-4834-bb77-d40b39b79696/volumes" Oct 30 19:26:44.414369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:44.414335 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:44.414939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:44.414908 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:45.414229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:45.413510 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:45.414229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:45.413578 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:46.412232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:46.404576 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:46.412232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:46.404638 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:47.404335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:47.403818 4789 patch_prober.go:29] interesting pod/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:26:47.404335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:47.403888 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:26:48.196132 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:26:48.196091 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:26:48.400608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:48.400582 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:26:48.401021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:48.400996 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:26:48.401743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:48.401722 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:26:49.971978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.971944 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr"] Oct 30 19:26:49.972606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972588 4789 cpu_manager.go:414] "containerMap: removing container" podUID="531ca2bb-b43c-47d5-a92b-0359afcef00f" containerName="revoke-keys" Oct 30 19:26:49.972656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972608 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531ca2bb-b43c-47d5-a92b-0359afcef00f" containerName="revoke-keys" Oct 30 19:26:49.972656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972626 4789 cpu_manager.go:414] "containerMap: removing container" podUID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" containerName="payload-processing" Oct 30 19:26:49.972656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972631 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" containerName="payload-processing" Oct 30 19:26:49.972761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972715 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="531ca2bb-b43c-47d5-a92b-0359afcef00f" containerName="revoke-keys" Oct 30 19:26:49.972761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.972724 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="318bd54a-e1eb-4fd2-81ab-d767efe33c63" containerName="payload-processing" Oct 30 19:26:49.977271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:49.977254 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.006108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.006086 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.006231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.006119 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zhs\" (UniqueName: \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.006339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.006321 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.031932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.031906 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr"] Oct 30 19:26:50.074023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.073989 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:26:50.074241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.074007 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:26:50.074241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.074125 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:26:50.074413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.074136 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:26:50.107849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.107825 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.107966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.107937 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.108035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.107975 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zhs\" (UniqueName: \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.108657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.108637 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.110135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.110110 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.165513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.165493 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zhs\" (UniqueName: \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\") pod \"maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.291101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.291052 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:50.692295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:50.692247 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr"] Oct 30 19:26:50.699127 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:26:50.699096 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc029c9b7_1a40_4329_abad_8ae80abfa8aa.slice/crio-17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64 WatchSource:0}: Error finding container 17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64: Status 404 returned error can't find the container with id 17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64 Oct 30 19:26:51.420946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:51.420854 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" event={"ID":"c029c9b7-1a40-4329-abad-8ae80abfa8aa","Type":"ContainerStarted","Data":"305d7986b4676b5c60be9e63da3b5ab128de4b294f3acc42cdc3a54a8c3419f8"} Oct 30 19:26:51.420946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:51.420919 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" event={"ID":"c029c9b7-1a40-4329-abad-8ae80abfa8aa","Type":"ContainerStarted","Data":"17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64"} Oct 30 19:26:51.531621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:51.531570 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" podStartSLOduration=2.531556797 podStartE2EDuration="2.531556797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:51.53090689 +0000 UTC m=+2448.032817389" watchObservedRunningTime="2026-10-30 19:26:51.531556797 +0000 UTC m=+2448.033467283" Oct 30 19:26:52.429781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:52.429741 4789 generic.go:356] "Generic (PLEG): container finished" podID="c029c9b7-1a40-4329-abad-8ae80abfa8aa" containerID="305d7986b4676b5c60be9e63da3b5ab128de4b294f3acc42cdc3a54a8c3419f8" exitCode=0 Oct 30 19:26:52.430266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:52.429829 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" event={"ID":"c029c9b7-1a40-4329-abad-8ae80abfa8aa","Type":"ContainerDied","Data":"305d7986b4676b5c60be9e63da3b5ab128de4b294f3acc42cdc3a54a8c3419f8"} Oct 30 19:26:53.605415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.605386 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:53.647615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.647593 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\") pod \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " Oct 30 19:26:53.647798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.647780 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\" (UniqueName: \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\") pod \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " Oct 30 19:26:53.647855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.647841 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\") pod \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\" (UID: \"c029c9b7-1a40-4329-abad-8ae80abfa8aa\") " Oct 30 19:26:53.648181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.648159 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca" pod "c029c9b7-1a40-4329-abad-8ae80abfa8aa" (UID: "c029c9b7-1a40-4329-abad-8ae80abfa8aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:53.649599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.649580 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca" pod "c029c9b7-1a40-4329-abad-8ae80abfa8aa" (UID: "c029c9b7-1a40-4329-abad-8ae80abfa8aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:53.649717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.649697 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs" pod "c029c9b7-1a40-4329-abad-8ae80abfa8aa" (UID: "c029c9b7-1a40-4329-abad-8ae80abfa8aa"). InnerVolumeSpecName "kube-api-access-79zhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:53.749624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.749551 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zhs\" (UniqueName: \"kubernetes.io/projected/c029c9b7-1a40-4329-abad-8ae80abfa8aa-kube-api-access-79zhs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:53.749624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.749581 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:53.749624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:53.749591 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c029c9b7-1a40-4329-abad-8ae80abfa8aa-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:54.219623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:54.219585 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 19:26:54.219850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:54.219824 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" containerID="cri-o://d511ea738e3843c741cd686a74f85ac31750646e950c51a552988a576b18276e" gracePeriod=30 Oct 30 19:26:54.445353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:54.445318 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" Oct 30 19:26:54.445353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:54.445342 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr" event={"ID":"c029c9b7-1a40-4329-abad-8ae80abfa8aa","Type":"ContainerDied","Data":"17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64"} Oct 30 19:26:54.445555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:54.445390 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a3aad294dce6322d4ac541ce03d738123714ca2ef7e44c5fb8416d0c141b64" Oct 30 19:26:59.481662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.481586 4789 generic.go:356] "Generic (PLEG): container finished" podID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerID="d511ea738e3843c741cd686a74f85ac31750646e950c51a552988a576b18276e" exitCode=0 Oct 30 19:26:59.482186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.481736 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" event={"ID":"74a47b28-6fdf-49e6-8705-4ca225fdcdbd","Type":"ContainerDied","Data":"d511ea738e3843c741cd686a74f85ac31750646e950c51a552988a576b18276e"} Oct 30 19:26:59.548028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.548006 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 19:26:59.651824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.651796 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.651962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.651903 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.651960 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652006 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652052 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652100 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:59.652299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652165 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652203 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652305 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652351 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\") pod \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\" (UID: \"74a47b28-6fdf-49e6-8705-4ca225fdcdbd\") " Oct 30 19:26:59.652538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652303 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:59.652538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652442 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:59.652538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652455 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:59.652843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652810 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.652843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652836 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.652978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652851 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.652978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652865 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.652978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.652919 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:59.654258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.654234 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:59.654507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.654486 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:26:59.654570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.654530 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:59.654665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.654648 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll" pod "74a47b28-6fdf-49e6-8705-4ca225fdcdbd" (UID: "74a47b28-6fdf-49e6-8705-4ca225fdcdbd"). InnerVolumeSpecName "kube-api-access-f6xll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:59.753838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.753776 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.753838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.753802 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.753838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.753812 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.753838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.753823 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xll\" (UniqueName: \"kubernetes.io/projected/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-kube-api-access-f6xll\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:59.753838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:26:59.753833 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74a47b28-6fdf-49e6-8705-4ca225fdcdbd-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:00.490697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:00.490661 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" Oct 30 19:27:00.490697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:00.490680 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz" event={"ID":"74a47b28-6fdf-49e6-8705-4ca225fdcdbd","Type":"ContainerDied","Data":"79810c3afda484c68391d8ba641e112b72b3225b09671e177a5813c0bdb2abdb"} Oct 30 19:27:00.491233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:00.490758 4789 scope.go:122] "RemoveContainer" containerID="d511ea738e3843c741cd686a74f85ac31750646e950c51a552988a576b18276e" Oct 30 19:27:00.650096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:00.650064 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 19:27:00.696041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:00.696016 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-p2prz"] Oct 30 19:27:01.384108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:01.384076 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:27:01.670272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:01.670175 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" path="/var/lib/kubelet/pods/74a47b28-6fdf-49e6-8705-4ca225fdcdbd/volumes" Oct 30 19:27:06.384954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.384838 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" podUID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" containerName="payload-processing" containerID="cri-o://1b83e0ab4ba451d444ca7c127773daf0165099379e9f055dbd9ca66cb139b063" gracePeriod=25 Oct 30 19:27:06.538831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.538793 4789 generic.go:356] "Generic (PLEG): container finished" podID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" containerID="1b83e0ab4ba451d444ca7c127773daf0165099379e9f055dbd9ca66cb139b063" exitCode=0 Oct 30 19:27:06.539006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.538875 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" event={"ID":"abfb9d87-7a69-480b-a9ad-ab8ca2351164","Type":"ContainerDied","Data":"1b83e0ab4ba451d444ca7c127773daf0165099379e9f055dbd9ca66cb139b063"} Oct 30 19:27:06.664655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.664632 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:27:06.739062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.739032 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\" (UniqueName: \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\") pod \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " Oct 30 19:27:06.739220 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.739099 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\") pod \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\" (UID: \"abfb9d87-7a69-480b-a9ad-ab8ca2351164\") " Oct 30 19:27:06.739725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.739560 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume" pod "abfb9d87-7a69-480b-a9ad-ab8ca2351164" (UID: "abfb9d87-7a69-480b-a9ad-ab8ca2351164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:06.741719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.741696 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp" pod "abfb9d87-7a69-480b-a9ad-ab8ca2351164" (UID: "abfb9d87-7a69-480b-a9ad-ab8ca2351164"). InnerVolumeSpecName "kube-api-access-msvvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:06.840427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.840392 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvvp\" (UniqueName: \"kubernetes.io/projected/abfb9d87-7a69-480b-a9ad-ab8ca2351164-kube-api-access-msvvp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:06.840427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:06.840416 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abfb9d87-7a69-480b-a9ad-ab8ca2351164-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:07.548364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:07.548337 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" Oct 30 19:27:07.548364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:07.548346 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g" event={"ID":"abfb9d87-7a69-480b-a9ad-ab8ca2351164","Type":"ContainerDied","Data":"f1f1d2c1abdd68d382c086fce6e2aebbd9a9e85dd3266295ac4478e88bc6c439"} Oct 30 19:27:07.548893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:07.548405 4789 scope.go:122] "RemoveContainer" containerID="1b83e0ab4ba451d444ca7c127773daf0165099379e9f055dbd9ca66cb139b063" Oct 30 19:27:07.709754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:07.709721 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:27:07.714905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:07.714881 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d6b5ac-846d6b5cf4-vj58g"] Oct 30 19:27:09.151724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:09.151691 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w"] Oct 30 19:27:09.197958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:09.197912 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ec347-5h28w"] Oct 30 19:27:09.667344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:09.667314 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531ca2bb-b43c-47d5-a92b-0359afcef00f" path="/var/lib/kubelet/pods/531ca2bb-b43c-47d5-a92b-0359afcef00f/volumes" Oct 30 19:27:09.667730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:09.667715 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" path="/var/lib/kubelet/pods/abfb9d87-7a69-480b-a9ad-ab8ca2351164/volumes" Oct 30 19:27:18.197592 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:27:18.197511 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:27:18.555598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.555517 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj"] Oct 30 19:27:18.556095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556076 4789 cpu_manager.go:414] "containerMap: removing container" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" Oct 30 19:27:18.556095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556095 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" Oct 30 19:27:18.556250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556129 4789 cpu_manager.go:414] "containerMap: removing container" podUID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" containerName="payload-processing" Oct 30 19:27:18.556250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556134 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" containerName="payload-processing" Oct 30 19:27:18.556250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556151 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c029c9b7-1a40-4329-abad-8ae80abfa8aa" containerName="revoke-keys" Oct 30 19:27:18.556250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556157 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c029c9b7-1a40-4329-abad-8ae80abfa8aa" containerName="revoke-keys" Oct 30 19:27:18.556374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556253 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c029c9b7-1a40-4329-abad-8ae80abfa8aa" containerName="revoke-keys" Oct 30 19:27:18.556374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556265 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a47b28-6fdf-49e6-8705-4ca225fdcdbd" containerName="istio-proxy" Oct 30 19:27:18.556374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.556275 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfb9d87-7a69-480b-a9ad-ab8ca2351164" containerName="payload-processing" Oct 30 19:27:18.560718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.560701 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.615609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.615537 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj"] Oct 30 19:27:18.657794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.657752 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:18.657794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.657776 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:27:18.657794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.657752 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:18.658089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.657783 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:18.671624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.671604 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.671732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.671635 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.671732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.671675 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbcl\" (UniqueName: \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.772361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.772320 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.772524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.772396 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbcl\" (UniqueName: \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.772603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.772569 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.773115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.773096 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.774857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.774836 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.830367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.830302 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbcl\" (UniqueName: \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\") pod \"maas-api-revoke-keys-e2e-ipp-b-539084-khwrj\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:18.873718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:18.873679 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:19.258255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:19.258220 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj"] Oct 30 19:27:19.264897 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:27:19.264862 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3904280_ab55_4c56_a99a_f073f674b3bf.slice/crio-edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc WatchSource:0}: Error finding container edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc: Status 404 returned error can't find the container with id edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc Oct 30 19:27:19.629168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:19.629136 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" event={"ID":"f3904280-ab55-4c56-a99a-f073f674b3bf","Type":"ContainerStarted","Data":"c1bc95c5a019432e6204cf6f4b65c8231067831a552af4778089dddfc1f0aca6"} Oct 30 19:27:19.629332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:19.629200 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" event={"ID":"f3904280-ab55-4c56-a99a-f073f674b3bf","Type":"ContainerStarted","Data":"edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc"} Oct 30 19:27:19.739818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:19.739767 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" podStartSLOduration=1.739752766 podStartE2EDuration="1.739752766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:19.738722173 +0000 UTC m=+2476.240632660" watchObservedRunningTime="2026-10-30 19:27:19.739752766 +0000 UTC m=+2476.241663252" Oct 30 19:27:20.637667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:20.637632 4789 generic.go:356] "Generic (PLEG): container finished" podID="f3904280-ab55-4c56-a99a-f073f674b3bf" containerID="c1bc95c5a019432e6204cf6f4b65c8231067831a552af4778089dddfc1f0aca6" exitCode=0 Oct 30 19:27:20.638126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:20.637717 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" event={"ID":"f3904280-ab55-4c56-a99a-f073f674b3bf","Type":"ContainerDied","Data":"c1bc95c5a019432e6204cf6f4b65c8231067831a552af4778089dddfc1f0aca6"} Oct 30 19:27:21.644307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.644277 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:27:21.644721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.644593 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" containerID="cri-o://7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643" gracePeriod=30 Oct 30 19:27:21.836238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.836196 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:21.906037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.905963 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\") pod \"f3904280-ab55-4c56-a99a-f073f674b3bf\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " Oct 30 19:27:21.906179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.906058 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\" (UniqueName: \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\") pod \"f3904280-ab55-4c56-a99a-f073f674b3bf\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " Oct 30 19:27:21.906179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.906168 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\") pod \"f3904280-ab55-4c56-a99a-f073f674b3bf\" (UID: \"f3904280-ab55-4c56-a99a-f073f674b3bf\") " Oct 30 19:27:21.906401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.906374 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca" pod "f3904280-ab55-4c56-a99a-f073f674b3bf" (UID: "f3904280-ab55-4c56-a99a-f073f674b3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:21.906576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.906553 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:21.908715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.908692 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca" pod "f3904280-ab55-4c56-a99a-f073f674b3bf" (UID: "f3904280-ab55-4c56-a99a-f073f674b3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:21.908830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:21.908729 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl" pod "f3904280-ab55-4c56-a99a-f073f674b3bf" (UID: "f3904280-ab55-4c56-a99a-f073f674b3bf"). InnerVolumeSpecName "kube-api-access-pvbcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:22.007165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.007132 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3904280-ab55-4c56-a99a-f073f674b3bf-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:22.007165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.007157 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbcl\" (UniqueName: \"kubernetes.io/projected/f3904280-ab55-4c56-a99a-f073f674b3bf-kube-api-access-pvbcl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:22.652141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.652112 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" Oct 30 19:27:22.652141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.652126 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj" event={"ID":"f3904280-ab55-4c56-a99a-f073f674b3bf","Type":"ContainerDied","Data":"edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc"} Oct 30 19:27:22.652677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.652170 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edaf14df30064d4d143d19ba1ce6692432dec85306b847ac1c01fdec40148ccc" Oct 30 19:27:22.990429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.990402 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:27:22.990625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:22.990584 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" containerID="cri-o://d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6" gracePeriod=30 Oct 30 19:27:23.146342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:23.146310 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:27:23.146496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:23.146359 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:27:25.453452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:25.453418 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:27:25.453925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:25.453477 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:27:26.953896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:26.953872 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:27:27.071131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071107 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071155 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071187 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071232 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071316 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071367 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:27.071534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071422 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071442 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:27.071534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071460 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071505 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071542 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\") pod \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\" (UID: \"b8945dc4-58b8-4998-aa6b-48d548bd53a1\") " Oct 30 19:27:27.071749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071667 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:27.071749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071686 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:27.071749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.071728 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:27.072135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.072032 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.072135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.072056 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.072135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.072071 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.072135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.072084 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.072135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.072099 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.073524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.073495 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:27.073656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.073636 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:27.073706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.073642 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "kube-api-access-s7sx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:27.073823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.073805 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo" pod "b8945dc4-58b8-4998-aa6b-48d548bd53a1" (UID: "b8945dc4-58b8-4998-aa6b-48d548bd53a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:27:27.173550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.173522 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7sx7\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-kube-api-access-s7sx7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.173550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.173547 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.173729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.173556 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.173729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.173566 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8945dc4-58b8-4998-aa6b-48d548bd53a1-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.703367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.703327 4789 generic.go:356] "Generic (PLEG): container finished" podID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerID="7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643" exitCode=0 Oct 30 19:27:27.703521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.703391 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" Oct 30 19:27:27.703521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.703412 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" event={"ID":"b8945dc4-58b8-4998-aa6b-48d548bd53a1","Type":"ContainerDied","Data":"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643"} Oct 30 19:27:27.703521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.703494 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn" event={"ID":"b8945dc4-58b8-4998-aa6b-48d548bd53a1","Type":"ContainerDied","Data":"bf5b7adae87bb7811da0bb74b64e10b286d90a6bc259b580fd71906f2ab3f178"} Oct 30 19:27:27.703521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.703517 4789 scope.go:122] "RemoveContainer" containerID="7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643" Oct 30 19:27:27.712869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.712850 4789 scope.go:122] "RemoveContainer" containerID="7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643" Oct 30 19:27:27.713109 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:27:27.713092 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643\": container with ID starting with 7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643 not found: ID does not exist" containerID="7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643" Oct 30 19:27:27.713164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.713116 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643"} err="failed to get container status \"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643\": rpc error: code = NotFound desc = could not find container \"7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643\": container with ID starting with 7c0fb333e4c385df4986227136213840f799276500d56b5fe3baf47950a85643 not found: ID does not exist" Oct 30 19:27:27.905267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.905240 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:27:27.951358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:27.951336 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-64hfn"] Oct 30 19:27:28.259538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.259511 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:27:28.385066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.384979 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385131 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385174 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385270 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385300 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385400 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385444 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385474 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\") pod \"d8410804-6293-453e-8fb0-62e1fa943e68\" (UID: \"d8410804-6293-453e-8fb0-62e1fa943e68\") " Oct 30 19:27:28.385710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385605 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:28.385913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385893 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:28.385993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.385950 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.386298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.386114 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:28.386298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.386127 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:28.386477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.386305 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:28.388723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.388688 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:28.388826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.388725 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:28.389632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.389600 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "kube-api-access-qfgqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:28.389746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.389655 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo" pod "d8410804-6293-453e-8fb0-62e1fa943e68" (UID: "d8410804-6293-453e-8fb0-62e1fa943e68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:27:28.487299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487230 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgqv\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-kube-api-access-qfgqv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487265 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487279 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487294 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487309 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8410804-6293-453e-8fb0-62e1fa943e68-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487322 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8410804-6293-453e-8fb0-62e1fa943e68-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487335 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8410804-6293-453e-8fb0-62e1fa943e68-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.487531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.487348 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8410804-6293-453e-8fb0-62e1fa943e68-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:28.712138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.712108 4789 generic.go:356] "Generic (PLEG): container finished" podID="d8410804-6293-453e-8fb0-62e1fa943e68" containerID="d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6" exitCode=0 Oct 30 19:27:28.712296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.712168 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" Oct 30 19:27:28.712296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.712188 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" event={"ID":"d8410804-6293-453e-8fb0-62e1fa943e68","Type":"ContainerDied","Data":"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6"} Oct 30 19:27:28.712296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.712251 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w" event={"ID":"d8410804-6293-453e-8fb0-62e1fa943e68","Type":"ContainerDied","Data":"96af1ec487fe644942705807f85072f945557007b89dadadc00c90a080119dbb"} Oct 30 19:27:28.712296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.712279 4789 scope.go:122] "RemoveContainer" containerID="d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6" Oct 30 19:27:28.722800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.722779 4789 scope.go:122] "RemoveContainer" containerID="d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6" Oct 30 19:27:28.723043 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:27:28.723019 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6\": container with ID starting with d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6 not found: ID does not exist" containerID="d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6" Oct 30 19:27:28.723126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.723053 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6"} err="failed to get container status \"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6\": rpc error: code = NotFound desc = could not find container \"d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6\": container with ID starting with d085cde6555791f940817bbb4d933373989692a3199095b8c6589b63e6d25ee6 not found: ID does not exist" Oct 30 19:27:28.873522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.873453 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:27:28.920070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:28.920042 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-zhm9w"] Oct 30 19:27:29.668164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:29.668125 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" path="/var/lib/kubelet/pods/b8945dc4-58b8-4998-aa6b-48d548bd53a1/volumes" Oct 30 19:27:29.668602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:29.668587 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" path="/var/lib/kubelet/pods/d8410804-6293-453e-8fb0-62e1fa943e68/volumes" Oct 30 19:27:31.058612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:31.058582 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:27:36.059565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.059527 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" podUID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" containerName="payload-processing" containerID="cri-o://1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b" gracePeriod=25 Oct 30 19:27:36.325892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.325830 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:27:36.482295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.482254 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\") pod \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " Oct 30 19:27:36.482447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.482304 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\" (UniqueName: \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\") pod \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\" (UID: \"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344\") " Oct 30 19:27:36.482718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.482690 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume" pod "d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" (UID: "d2b7f3bb-4b97-40e9-a6fb-30e813cd9344"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:36.484386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.484363 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq" pod "d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" (UID: "d2b7f3bb-4b97-40e9-a6fb-30e813cd9344"). InnerVolumeSpecName "kube-api-access-wz7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:36.583968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.583901 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:36.583968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.583928 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7pq\" (UniqueName: \"kubernetes.io/projected/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344-kube-api-access-wz7pq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:36.760997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.760967 4789 generic.go:356] "Generic (PLEG): container finished" podID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" containerID="1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b" exitCode=0 Oct 30 19:27:36.761164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.761011 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" Oct 30 19:27:36.761164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.761059 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" event={"ID":"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344","Type":"ContainerDied","Data":"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b"} Oct 30 19:27:36.761164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.761110 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd" event={"ID":"d2b7f3bb-4b97-40e9-a6fb-30e813cd9344","Type":"ContainerDied","Data":"dde28101b00d1710642393ae284cc290d73a966aa7f99573c53fb0a9d0e85da8"} Oct 30 19:27:36.761164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.761133 4789 scope.go:122] "RemoveContainer" containerID="1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b" Oct 30 19:27:36.771176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.771156 4789 scope.go:122] "RemoveContainer" containerID="1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b" Oct 30 19:27:36.771445 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:27:36.771420 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b\": container with ID starting with 1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b not found: ID does not exist" containerID="1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b" Oct 30 19:27:36.771509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.771453 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b"} err="failed to get container status \"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b\": rpc error: code = NotFound desc = could not find container \"1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b\": container with ID starting with 1caecf4b1446d6a8adc547cadd110f897d99caa21349df94ba0d9dfd0b82823b not found: ID does not exist" Oct 30 19:27:36.918834 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.918757 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:27:36.923884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:36.923860 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2a5c13-bdb8dc5b4-hwzvd"] Oct 30 19:27:37.618694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.618660 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj"] Oct 30 19:27:37.619200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619180 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619201 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619253 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" containerName="payload-processing" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619259 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" containerName="payload-processing" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619272 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619277 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619287 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f3904280-ab55-4c56-a99a-f073f674b3bf" containerName="revoke-keys" Oct 30 19:27:37.619292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619292 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3904280-ab55-4c56-a99a-f073f674b3bf" containerName="revoke-keys" Oct 30 19:27:37.619653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619382 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8945dc4-58b8-4998-aa6b-48d548bd53a1" containerName="istio-proxy" Oct 30 19:27:37.619653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619393 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" containerName="payload-processing" Oct 30 19:27:37.619653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619400 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8410804-6293-453e-8fb0-62e1fa943e68" containerName="istio-proxy" Oct 30 19:27:37.619653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.619406 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3904280-ab55-4c56-a99a-f073f674b3bf" containerName="revoke-keys" Oct 30 19:27:37.624066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.624041 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.668537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.668516 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b7f3bb-4b97-40e9-a6fb-30e813cd9344" path="/var/lib/kubelet/pods/d2b7f3bb-4b97-40e9-a6fb-30e813cd9344/volumes" Oct 30 19:27:37.675378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.675358 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:27:37.675662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.675627 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:27:37.675778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.675681 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:37.679524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.679499 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj"] Oct 30 19:27:37.695514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.695490 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.695744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.695717 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2fx\" (UniqueName: \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.695909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.695882 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.722231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.721980 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:37.796627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.796589 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2fx\" (UniqueName: \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.796763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.796660 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.796763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.796714 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.797448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.797425 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.799035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.799012 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.854953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.854921 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2fx\" (UniqueName: \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\") pod \"maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:37.941478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:37.941452 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:38.318954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:38.318864 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj"] Oct 30 19:27:38.325718 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:27:38.325686 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd792164_14b7_4f65_bd57_2ba57fc2d215.slice/crio-9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9 WatchSource:0}: Error finding container 9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9: Status 404 returned error can't find the container with id 9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9 Oct 30 19:27:38.776868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:38.776841 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" event={"ID":"dd792164-14b7-4f65-bd57-2ba57fc2d215","Type":"ContainerStarted","Data":"d505911d074246b7fba18992005e30d98c71820859f9d2367ae4292bd9dbb6b0"} Oct 30 19:27:38.777200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:38.776897 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" event={"ID":"dd792164-14b7-4f65-bd57-2ba57fc2d215","Type":"ContainerStarted","Data":"9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9"} Oct 30 19:27:38.888075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:38.888032 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" podStartSLOduration=1.8880149130000001 podStartE2EDuration="1.888014913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:38.886845639 +0000 UTC m=+2495.388756129" watchObservedRunningTime="2026-10-30 19:27:38.888014913 +0000 UTC m=+2495.389925398" Oct 30 19:27:39.784127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:39.784092 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd792164-14b7-4f65-bd57-2ba57fc2d215" containerID="d505911d074246b7fba18992005e30d98c71820859f9d2367ae4292bd9dbb6b0" exitCode=0 Oct 30 19:27:39.784595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:39.784176 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" event={"ID":"dd792164-14b7-4f65-bd57-2ba57fc2d215","Type":"ContainerDied","Data":"d505911d074246b7fba18992005e30d98c71820859f9d2367ae4292bd9dbb6b0"} Oct 30 19:27:40.926892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:40.926865 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:41.034503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.034474 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\" (UniqueName: \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\") pod \"dd792164-14b7-4f65-bd57-2ba57fc2d215\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " Oct 30 19:27:41.034697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.034557 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\") pod \"dd792164-14b7-4f65-bd57-2ba57fc2d215\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " Oct 30 19:27:41.034697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.034579 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\") pod \"dd792164-14b7-4f65-bd57-2ba57fc2d215\" (UID: \"dd792164-14b7-4f65-bd57-2ba57fc2d215\") " Oct 30 19:27:41.034992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.034966 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca" pod "dd792164-14b7-4f65-bd57-2ba57fc2d215" (UID: "dd792164-14b7-4f65-bd57-2ba57fc2d215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:41.036719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.036677 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx" pod "dd792164-14b7-4f65-bd57-2ba57fc2d215" (UID: "dd792164-14b7-4f65-bd57-2ba57fc2d215"). InnerVolumeSpecName "kube-api-access-gd2fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:41.036830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.036761 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca" pod "dd792164-14b7-4f65-bd57-2ba57fc2d215" (UID: "dd792164-14b7-4f65-bd57-2ba57fc2d215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:41.136185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.136149 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:41.136185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.136181 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd792164-14b7-4f65-bd57-2ba57fc2d215-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:41.136185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.136191 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2fx\" (UniqueName: \"kubernetes.io/projected/dd792164-14b7-4f65-bd57-2ba57fc2d215-kube-api-access-gd2fx\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:41.797321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.797293 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" Oct 30 19:27:41.797472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.797332 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj" event={"ID":"dd792164-14b7-4f65-bd57-2ba57fc2d215","Type":"ContainerDied","Data":"9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9"} Oct 30 19:27:41.797472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:27:41.797378 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d04a6a98719d226fe7657e90d05474e422e64a00d3039d79e5200fd6d60afd9" Oct 30 19:27:48.195985 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:27:48.195944 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:28:13.327572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.327539 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:28:13.328051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.327753 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" podUID="606895ee-8976-4648-b97d-8ce9d917efad" containerName="payload-pre-processing" containerID="cri-o://1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d" gracePeriod=30 Oct 30 19:28:13.589080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.589025 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:28:13.676076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.676045 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\") pod \"606895ee-8976-4648-b97d-8ce9d917efad\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " Oct 30 19:28:13.676256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.676156 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\" (UniqueName: \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\") pod \"606895ee-8976-4648-b97d-8ce9d917efad\" (UID: \"606895ee-8976-4648-b97d-8ce9d917efad\") " Oct 30 19:28:13.676467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.676444 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume" pod "606895ee-8976-4648-b97d-8ce9d917efad" (UID: "606895ee-8976-4648-b97d-8ce9d917efad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:13.677136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.677095 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606895ee-8976-4648-b97d-8ce9d917efad-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:13.686031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.685995 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22" pod "606895ee-8976-4648-b97d-8ce9d917efad" (UID: "606895ee-8976-4648-b97d-8ce9d917efad"). InnerVolumeSpecName "kube-api-access-zfc22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:13.778305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:13.778277 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfc22\" (UniqueName: \"kubernetes.io/projected/606895ee-8976-4648-b97d-8ce9d917efad-kube-api-access-zfc22\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.016560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.016523 4789 generic.go:356] "Generic (PLEG): container finished" podID="606895ee-8976-4648-b97d-8ce9d917efad" containerID="1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d" exitCode=0 Oct 30 19:28:14.016749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.016584 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" Oct 30 19:28:14.016749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.016609 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" event={"ID":"606895ee-8976-4648-b97d-8ce9d917efad","Type":"ContainerDied","Data":"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d"} Oct 30 19:28:14.016749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.016702 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv" event={"ID":"606895ee-8976-4648-b97d-8ce9d917efad","Type":"ContainerDied","Data":"e015d21653fdc0e1e51217bd523a6cd09602922ea4912cb3e979adb8ea8ce813"} Oct 30 19:28:14.016749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.016733 4789 scope.go:122] "RemoveContainer" containerID="1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d" Oct 30 19:28:14.027068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.027049 4789 scope.go:122] "RemoveContainer" containerID="1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d" Oct 30 19:28:14.027342 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:28:14.027320 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d\": container with ID starting with 1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d not found: ID does not exist" containerID="1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d" Oct 30 19:28:14.027413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.027350 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d"} err="failed to get container status \"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d\": rpc error: code = NotFound desc = could not find container \"1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d\": container with ID starting with 1b833043f1ec90cbada7e95f6878b3fea666288ab7d32debbbc425a4f5291b8d not found: ID does not exist" Oct 30 19:28:14.172962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.172938 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:28:14.178822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:14.178803 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-539084-84b5c6d79f-dswmv"] Oct 30 19:28:15.020514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:15.020483 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj"] Oct 30 19:28:15.066549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:15.066519 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-539084-khwrj"] Oct 30 19:28:15.668876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:15.668841 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606895ee-8976-4648-b97d-8ce9d917efad" path="/var/lib/kubelet/pods/606895ee-8976-4648-b97d-8ce9d917efad/volumes" Oct 30 19:28:15.669460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:15.669443 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3904280-ab55-4c56-a99a-f073f674b3bf" path="/var/lib/kubelet/pods/f3904280-ab55-4c56-a99a-f073f674b3bf/volumes" Oct 30 19:28:18.196351 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:28:18.196316 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:28:26.256465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.256432 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:28:26.256979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.256694 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" podUID="ea13fb92-b3c3-4857-8401-4031048352e3" containerName="payload-pre-processing" containerID="cri-o://0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de" gracePeriod=30 Oct 30 19:28:26.548512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.548479 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:28:26.617602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.617566 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\") pod \"ea13fb92-b3c3-4857-8401-4031048352e3\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " Oct 30 19:28:26.617799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.617686 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\" (UniqueName: \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\") pod \"ea13fb92-b3c3-4857-8401-4031048352e3\" (UID: \"ea13fb92-b3c3-4857-8401-4031048352e3\") " Oct 30 19:28:26.618778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.618747 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume" pod "ea13fb92-b3c3-4857-8401-4031048352e3" (UID: "ea13fb92-b3c3-4857-8401-4031048352e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:26.624803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.624757 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw" pod "ea13fb92-b3c3-4857-8401-4031048352e3" (UID: "ea13fb92-b3c3-4857-8401-4031048352e3"). InnerVolumeSpecName "kube-api-access-pkrfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:26.725933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.719367 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrfw\" (UniqueName: \"kubernetes.io/projected/ea13fb92-b3c3-4857-8401-4031048352e3-kube-api-access-pkrfw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:26.725933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:26.719400 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea13fb92-b3c3-4857-8401-4031048352e3-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:27.098465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.098427 4789 generic.go:356] "Generic (PLEG): container finished" podID="ea13fb92-b3c3-4857-8401-4031048352e3" containerID="0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de" exitCode=0 Oct 30 19:28:27.098638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.098474 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" Oct 30 19:28:27.098638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.098522 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" event={"ID":"ea13fb92-b3c3-4857-8401-4031048352e3","Type":"ContainerDied","Data":"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de"} Oct 30 19:28:27.098638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.098590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc" event={"ID":"ea13fb92-b3c3-4857-8401-4031048352e3","Type":"ContainerDied","Data":"028c71dd0d55a76337581b3cdefecf2090be86e2e50807d5da88a60d9a0f315c"} Oct 30 19:28:27.098638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.098619 4789 scope.go:122] "RemoveContainer" containerID="0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de" Oct 30 19:28:27.111465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.111440 4789 scope.go:122] "RemoveContainer" containerID="0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de" Oct 30 19:28:27.111780 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:28:27.111754 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de\": container with ID starting with 0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de not found: ID does not exist" containerID="0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de" Oct 30 19:28:27.111862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.111792 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de"} err="failed to get container status \"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de\": rpc error: code = NotFound desc = could not find container \"0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de\": container with ID starting with 0aaeb73895cfd575da7911c649da40a6e327baa2b467ed6811e4cc718d7a60de not found: ID does not exist" Oct 30 19:28:27.269428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.269401 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:28:27.272881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.272859 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-496c5b-56bcf87c67-j82sc"] Oct 30 19:28:27.668431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:27.668400 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea13fb92-b3c3-4857-8401-4031048352e3" path="/var/lib/kubelet/pods/ea13fb92-b3c3-4857-8401-4031048352e3/volumes" Oct 30 19:28:28.007482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:28.007445 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj"] Oct 30 19:28:28.022830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:28.022804 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-496c5b-42kjj"] Oct 30 19:28:29.667676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:29.667644 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd792164-14b7-4f65-bd57-2ba57fc2d215" path="/var/lib/kubelet/pods/dd792164-14b7-4f65-bd57-2ba57fc2d215/volumes" Oct 30 19:28:36.977948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:36.977915 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:28:36.978504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:36.978133 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" containerID="cri-o://3c52e5d8c82a782fe8732443d112b383e45046b3c455eb35bae3493de65599b8" gracePeriod=30 Oct 30 19:28:42.197651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.197619 4789 generic.go:356] "Generic (PLEG): container finished" podID="e86dc6b1-d110-4577-821c-9558190efbd4" containerID="3c52e5d8c82a782fe8732443d112b383e45046b3c455eb35bae3493de65599b8" exitCode=0 Oct 30 19:28:42.198107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.197708 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerDied","Data":"3c52e5d8c82a782fe8732443d112b383e45046b3c455eb35bae3493de65599b8"} Oct 30 19:28:42.198107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.197789 4789 scope.go:122] "RemoveContainer" containerID="72caa5b4452751ce38a10cd8d0308e4a4fae926e7b895a8ee862298be6476dd6" Oct 30 19:28:42.334599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.334578 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:28:42.494970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.494900 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.494967 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.494998 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495019 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495045 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495100 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495132 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495162 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495182 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:42.495421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495312 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:42.495421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\") pod \"e86dc6b1-d110-4577-821c-9558190efbd4\" (UID: \"e86dc6b1-d110-4577-821c-9558190efbd4\") " Oct 30 19:28:42.495726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495419 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:42.495726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495606 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:42.495726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495647 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:42.495854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495820 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.495854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495837 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e86dc6b1-d110-4577-821c-9558190efbd4-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.495953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495853 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.495953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495867 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.495953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.495881 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.497487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.497461 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:42.497785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.497759 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:42.498072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.498048 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:42.498072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.498064 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd" pod "e86dc6b1-d110-4577-821c-9558190efbd4" (UID: "e86dc6b1-d110-4577-821c-9558190efbd4"). InnerVolumeSpecName "kube-api-access-2znpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:42.596901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.596825 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e86dc6b1-d110-4577-821c-9558190efbd4-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.596901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.596852 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znpd\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-kube-api-access-2znpd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.596901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.596864 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e86dc6b1-d110-4577-821c-9558190efbd4-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:42.596901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:42.596874 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e86dc6b1-d110-4577-821c-9558190efbd4-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:43.206717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.206687 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" Oct 30 19:28:43.207186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.206714 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb" event={"ID":"e86dc6b1-d110-4577-821c-9558190efbd4","Type":"ContainerDied","Data":"dc721272af2e6a17a6a0038130c447465842f9d846012945ad758860f94d44dc"} Oct 30 19:28:43.207186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.206770 4789 scope.go:122] "RemoveContainer" containerID="3c52e5d8c82a782fe8732443d112b383e45046b3c455eb35bae3493de65599b8" Oct 30 19:28:43.367280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.367249 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:28:43.380868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.380837 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-496c5b-openshift-default-6c5899b5b7-wp4gb"] Oct 30 19:28:43.668769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:43.668693 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" path="/var/lib/kubelet/pods/e86dc6b1-d110-4577-821c-9558190efbd4/volumes" Oct 30 19:28:44.261232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261173 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:28:44.261825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261805 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd792164-14b7-4f65-bd57-2ba57fc2d215" containerName="revoke-keys" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261828 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd792164-14b7-4f65-bd57-2ba57fc2d215" containerName="revoke-keys" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261850 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261858 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261875 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ea13fb92-b3c3-4857-8401-4031048352e3" containerName="payload-pre-processing" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261884 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea13fb92-b3c3-4857-8401-4031048352e3" containerName="payload-pre-processing" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261901 4789 cpu_manager.go:414] "containerMap: removing container" podUID="606895ee-8976-4648-b97d-8ce9d917efad" containerName="payload-pre-processing" Oct 30 19:28:44.261913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261908 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606895ee-8976-4648-b97d-8ce9d917efad" containerName="payload-pre-processing" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261931 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.261939 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.262069 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd792164-14b7-4f65-bd57-2ba57fc2d215" containerName="revoke-keys" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.262085 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.262096 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea13fb92-b3c3-4857-8401-4031048352e3" containerName="payload-pre-processing" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.262107 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="606895ee-8976-4648-b97d-8ce9d917efad" containerName="payload-pre-processing" Oct 30 19:28:44.262296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.262118 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86dc6b1-d110-4577-821c-9558190efbd4" containerName="istio-proxy" Oct 30 19:28:44.266755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.266734 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.311888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.311860 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.312029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.311909 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6b8\" (UniqueName: \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.312029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.312006 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.312159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.312141 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.312202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.312176 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.312296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.312284 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.322765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.322742 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:28:44.366174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.366148 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7a790b\"/\"openshift-service-ca.crt\"" Oct 30 19:28:44.366308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.366177 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7a790b\"/\"kube-root-ca.crt\"" Oct 30 19:28:44.366500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.366485 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7a790b\"/\"ipp-route-7a790b-kserve-self-signed-certs\"" Oct 30 19:28:44.366627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.366515 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7a790b\"/\"default-dockercfg-6rd9b\"" Oct 30 19:28:44.412901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.412870 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.412923 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.412965 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6b8\" (UniqueName: \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.412994 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.413068 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.413089 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.413461 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.413538 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.413582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.413532 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.415291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.415269 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.415583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.415566 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.473592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.473563 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6b8\" (UniqueName: \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\") pod \"ipp-route-7a790b-kserve-788b76f67c-crszt\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.580048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.579955 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:44.974523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:44.974481 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:28:44.976217 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:28:44.976182 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd3cf2f5_f940_4ad7_b22a_7dffa35983d2.slice/crio-14e19ccc15eba63372c6111bab0002d5f795b9438184288d9d18b3ec68330b7a WatchSource:0}: Error finding container 14e19ccc15eba63372c6111bab0002d5f795b9438184288d9d18b3ec68330b7a: Status 404 returned error can't find the container with id 14e19ccc15eba63372c6111bab0002d5f795b9438184288d9d18b3ec68330b7a Oct 30 19:28:45.223863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:45.223788 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" event={"ID":"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2","Type":"ContainerStarted","Data":"4125b2b4e5b232bf12de95aee2894da5f3dada711a63f911698a451b20ca3247"} Oct 30 19:28:45.223863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:45.223831 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" event={"ID":"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2","Type":"ContainerStarted","Data":"14e19ccc15eba63372c6111bab0002d5f795b9438184288d9d18b3ec68330b7a"} Oct 30 19:28:45.339838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:45.339783 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" podStartSLOduration=1.339769567 podStartE2EDuration="1.339769567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:45.338073371 +0000 UTC m=+2561.839983857" watchObservedRunningTime="2026-10-30 19:28:45.339769567 +0000 UTC m=+2561.841680053" Oct 30 19:28:45.580291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:45.580193 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:46.800802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:46.800773 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr"] Oct 30 19:28:46.846655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:46.846628 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b5ac-5w8pr"] Oct 30 19:28:47.668226 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:47.668184 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c029c9b7-1a40-4329-abad-8ae80abfa8aa" path="/var/lib/kubelet/pods/c029c9b7-1a40-4329-abad-8ae80abfa8aa/volumes" Oct 30 19:28:48.196452 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:28:48.196368 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:28:54.580595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:54.580560 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:54.593024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:54.592998 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:55.293810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:55.293781 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:28:55.471792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:55.471754 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:28:55.472106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:28:55.472077 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" containerID="cri-o://07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360" gracePeriod=30 Oct 30 19:29:00.737649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.737590 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:29:00.904740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.904708 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.904907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.904764 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.904907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.904797 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905012 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:00.905263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905095 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905155 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:00.905350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905318 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:00.905350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905321 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905427 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905483 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905549 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.905784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905760 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:00.906063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.905578 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\") pod \"36cce3d0-2210-4a42-8c43-970429d4e7b1\" (UID: \"36cce3d0-2210-4a42-8c43-970429d4e7b1\") " Oct 30 19:29:00.906405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.906384 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:00.906533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.906408 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:00.906533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.906422 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:00.906533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.906419 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:00.906533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.906437 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:00.912669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.912645 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:29:00.912817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.912689 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "kube-api-access-9d2pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:00.912817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.912748 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:29:00.913109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:00.913090 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token" pod "36cce3d0-2210-4a42-8c43-970429d4e7b1" (UID: "36cce3d0-2210-4a42-8c43-970429d4e7b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:01.007122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.007070 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:01.007122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.007094 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:01.007122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.007104 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2pn\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-kube-api-access-9d2pn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:01.007122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.007114 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36cce3d0-2210-4a42-8c43-970429d4e7b1-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:01.007122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.007122 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36cce3d0-2210-4a42-8c43-970429d4e7b1-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:01.326682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.326619 4789 generic.go:356] "Generic (PLEG): container finished" podID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerID="07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360" exitCode=0 Oct 30 19:29:01.326682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.326666 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" Oct 30 19:29:01.326820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.326687 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerDied","Data":"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360"} Oct 30 19:29:01.326820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.326735 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx" event={"ID":"36cce3d0-2210-4a42-8c43-970429d4e7b1","Type":"ContainerDied","Data":"ed03ed9b9db15e08ec34dd79a4e0fd66fbfa439ac619b44774967a27d10031d3"} Oct 30 19:29:01.326820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.326755 4789 scope.go:122] "RemoveContainer" containerID="07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360" Oct 30 19:29:01.336646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.336630 4789 scope.go:122] "RemoveContainer" containerID="bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859" Oct 30 19:29:01.345328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.345311 4789 scope.go:122] "RemoveContainer" containerID="07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360" Oct 30 19:29:01.345559 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:29:01.345540 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360\": container with ID starting with 07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360 not found: ID does not exist" containerID="07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360" Oct 30 19:29:01.345611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.345567 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360"} err="failed to get container status \"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360\": rpc error: code = NotFound desc = could not find container \"07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360\": container with ID starting with 07e6e7b69a63dc77e3f7024ae322283f7bc685b80f918d3dfd55e9e34d45f360 not found: ID does not exist" Oct 30 19:29:01.345611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.345586 4789 scope.go:122] "RemoveContainer" containerID="bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859" Oct 30 19:29:01.345832 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:29:01.345813 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859\": container with ID starting with bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859 not found: ID does not exist" containerID="bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859" Oct 30 19:29:01.345874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.345838 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859"} err="failed to get container status \"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859\": rpc error: code = NotFound desc = could not find container \"bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859\": container with ID starting with bba7fef2be51fc04213e906439bcc16b743a24771f04e5a0213e5f2bd67b2859 not found: ID does not exist" Oct 30 19:29:01.512355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.512324 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:29:01.558712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.558683 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d6b5ac-openshift-default-6b547c795c-5smmx"] Oct 30 19:29:01.667692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:01.667608 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" path="/var/lib/kubelet/pods/36cce3d0-2210-4a42-8c43-970429d4e7b1/volumes" Oct 30 19:29:15.261984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.261952 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:29:15.262566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.262548 4789 cpu_manager.go:414] "containerMap: removing container" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.262619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.262568 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.262619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.262585 4789 cpu_manager.go:414] "containerMap: removing container" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.262619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.262590 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.262720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.262677 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.267661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.267636 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.316616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.316590 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f3b5af\"" Oct 30 19:29:15.362850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.362809 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f3b5af-dockercfg-98sbk\"" Oct 30 19:29:15.365295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.365276 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:29:15.367401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.367382 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.367572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.367554 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrdr\" (UniqueName: \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.423172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.423137 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:29:15.424034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.424012 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cce3d0-2210-4a42-8c43-970429d4e7b1" containerName="istio-proxy" Oct 30 19:29:15.427853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.427835 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.468821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.468798 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwf8d\" (UniqueName: \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.468939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.468845 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.469034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.469018 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrdr\" (UniqueName: \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.469115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.469102 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.469738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.469718 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.500761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.500734 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:29:15.534137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.534069 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrdr\" (UniqueName: \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\") pod \"payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.570280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.570252 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwf8d\" (UniqueName: \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.570387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.570300 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.570952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.570933 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.581040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.581018 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:15.628587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.628551 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwf8d\" (UniqueName: \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\") pod \"payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.741629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.741601 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:15.958976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:15.958951 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:29:15.960952 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:29:15.960928 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7c23c1_b903_4930_ae24_02b863c157cf.slice/crio-4b3c5743898279fd22ca2b42125f5f558e6c730d2b9cfe77b30026b37bf315c7 WatchSource:0}: Error finding container 4b3c5743898279fd22ca2b42125f5f558e6c730d2b9cfe77b30026b37bf315c7: Status 404 returned error can't find the container with id 4b3c5743898279fd22ca2b42125f5f558e6c730d2b9cfe77b30026b37bf315c7 Oct 30 19:29:16.111972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.111942 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:29:16.113014 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:29:16.112978 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8438e49_6f00_4dc7_8253_9ff738f80437.slice/crio-5ae068d8f636d55a3a1ad09e89d32d2c95e5c0ca7b71651b0c8ddb0167638821 WatchSource:0}: Error finding container 5ae068d8f636d55a3a1ad09e89d32d2c95e5c0ca7b71651b0c8ddb0167638821: Status 404 returned error can't find the container with id 5ae068d8f636d55a3a1ad09e89d32d2c95e5c0ca7b71651b0c8ddb0167638821 Oct 30 19:29:16.424971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.424888 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" event={"ID":"b8438e49-6f00-4dc7-8253-9ff738f80437","Type":"ContainerStarted","Data":"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375"} Oct 30 19:29:16.424971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.424936 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" event={"ID":"b8438e49-6f00-4dc7-8253-9ff738f80437","Type":"ContainerStarted","Data":"5ae068d8f636d55a3a1ad09e89d32d2c95e5c0ca7b71651b0c8ddb0167638821"} Oct 30 19:29:16.425503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.425083 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:16.426617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.426590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" event={"ID":"1b7c23c1-b903-4930-ae24-02b863c157cf","Type":"ContainerStarted","Data":"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50"} Oct 30 19:29:16.426756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.426635 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" event={"ID":"1b7c23c1-b903-4930-ae24-02b863c157cf","Type":"ContainerStarted","Data":"4b3c5743898279fd22ca2b42125f5f558e6c730d2b9cfe77b30026b37bf315c7"} Oct 30 19:29:16.426756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.426675 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:16.537855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.537813 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" podStartSLOduration=1.53780003 podStartE2EDuration="1.53780003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:16.535675275 +0000 UTC m=+2593.037585763" watchObservedRunningTime="2026-10-30 19:29:16.53780003 +0000 UTC m=+2593.039710573" Oct 30 19:29:16.645862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:16.645822 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" podStartSLOduration=1.645810945 podStartE2EDuration="1.645810945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:16.645599268 +0000 UTC m=+2593.147509771" watchObservedRunningTime="2026-10-30 19:29:16.645810945 +0000 UTC m=+2593.147721433" Oct 30 19:29:18.200118 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:29:18.200080 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:29:27.435957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:27.435927 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:29:27.436443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:29:27.436002 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:29:48.196024 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:29:48.195993 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:30:00.174877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.174829 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:00.178759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.178742 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.186093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.186065 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:00.189982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.189965 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:00.190078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.190065 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.194134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.194114 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.221157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.221132 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:00.225517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.225498 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:00.225614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.225595 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.229183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.229168 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.230796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.230779 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:30:00.233082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.233064 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:00.233157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.233130 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:00.267467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.267442 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:00.270944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.270926 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:00.271056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.271042 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.274575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.274557 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.282971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.282949 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:00.286707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.286688 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:00.293624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293603 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.293709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293634 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65w7\" (UniqueName: \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.293709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293679 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tq48\" (UniqueName: \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.293872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293854 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmm6\" (UniqueName: \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.293937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293899 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.293998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.293985 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.315687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.315547 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:00.320450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.320428 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:00.324290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.324266 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:00.330224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.330182 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:00.366721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.366686 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:00.394720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394694 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmm6\" (UniqueName: \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.394883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394728 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slnp\" (UniqueName: \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.394883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394751 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.394883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394833 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.394883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394855 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.394883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394879 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.395182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.394935 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.395182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82k6\" (UniqueName: \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.395182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395141 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.395182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395167 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65w7\" (UniqueName: \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.395458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395199 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdnl\" (UniqueName: \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.395458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395271 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmch4\" (UniqueName: \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.395458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395351 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tq48\" (UniqueName: \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.395751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395570 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.395751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395695 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.395867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395699 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.395923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.395913 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.452847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.452819 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65w7\" (UniqueName: \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.453097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.453074 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmm6\" (UniqueName: \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\") pod \"maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.453693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.453673 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tq48\" (UniqueName: \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.492823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.492801 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:00.497195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.496940 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.497195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.496966 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.497195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.496994 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.497195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497018 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82k6\" (UniqueName: \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.497195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497128 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdnl\" (UniqueName: \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.497519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497367 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmch4\" (UniqueName: \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.497519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497477 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.497636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497548 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slnp\" (UniqueName: \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.497838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497812 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.497995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.497857 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.498082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.498063 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.498413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.498390 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.504011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.503989 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:00.510727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.510704 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:00.560178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.559767 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82k6\" (UniqueName: \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.561230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.560888 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdnl\" (UniqueName: \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.564288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.564190 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slnp\" (UniqueName: \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\") pod \"maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.564288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.564241 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmch4\" (UniqueName: \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.585279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.584898 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:00.593740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.593550 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:00.839908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.839830 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:00.845955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.845932 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:00.875290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.875263 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:00.882361 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:00.882333 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b80746e_ded7_431a_b8f1_3e4f1713f31e.slice/crio-7a5382cd90ce6066a3e87fee880d4e3f90ea452093890ae42da59e00e533db05 WatchSource:0}: Error finding container 7a5382cd90ce6066a3e87fee880d4e3f90ea452093890ae42da59e00e533db05: Status 404 returned error can't find the container with id 7a5382cd90ce6066a3e87fee880d4e3f90ea452093890ae42da59e00e533db05 Oct 30 19:30:00.924095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.924065 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:00.927370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.927282 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:00.938869 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:00.938841 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b9e5cd_a841_42c8_a898_3c0206a42ed1.slice/crio-0be5539d014784f559b31297ac6e12c2866016fe05d6782b33a053bc688213a9 WatchSource:0}: Error finding container 0be5539d014784f559b31297ac6e12c2866016fe05d6782b33a053bc688213a9: Status 404 returned error can't find the container with id 0be5539d014784f559b31297ac6e12c2866016fe05d6782b33a053bc688213a9 Oct 30 19:30:00.939362 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:00.939334 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b3ee7d_f79a_4a99_bc48_5ea894290b36.slice/crio-5e83837880548ad3a92d3fbc21d7833e9539a5072238842eb971a642ef43d98b WatchSource:0}: Error finding container 5e83837880548ad3a92d3fbc21d7833e9539a5072238842eb971a642ef43d98b: Status 404 returned error can't find the container with id 5e83837880548ad3a92d3fbc21d7833e9539a5072238842eb971a642ef43d98b Oct 30 19:30:00.974174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:00.974133 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:00.987242 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:00.986082 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de2cfb0_4f3c_4684_b68f_8e87b985bd7a.slice/crio-c4f649f373a2c1ac917ffec160523b0da439b18439ebf2b43b4211c6a8ca91eb WatchSource:0}: Error finding container c4f649f373a2c1ac917ffec160523b0da439b18439ebf2b43b4211c6a8ca91eb: Status 404 returned error can't find the container with id c4f649f373a2c1ac917ffec160523b0da439b18439ebf2b43b4211c6a8ca91eb Oct 30 19:30:01.004437 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:01.004364 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee35881_5f88_4564_bc41_a50e1fbf40da.slice/crio-87591a0a1efe8e20bb126dabd095611dab1c406d95963a60eb2522a412317a53 WatchSource:0}: Error finding container 87591a0a1efe8e20bb126dabd095611dab1c406d95963a60eb2522a412317a53: Status 404 returned error can't find the container with id 87591a0a1efe8e20bb126dabd095611dab1c406d95963a60eb2522a412317a53 Oct 30 19:30:01.007661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.007641 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:01.233634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.233602 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:01.237863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.237837 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:01.240050 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:01.238288 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fbeb12_4da0_473b_b064_071f95348b7b.slice/crio-ed2662914a8379469eeabc7311bbe8fd8656f83300633483b1e62786f8564917 WatchSource:0}: Error finding container ed2662914a8379469eeabc7311bbe8fd8656f83300633483b1e62786f8564917: Status 404 returned error can't find the container with id ed2662914a8379469eeabc7311bbe8fd8656f83300633483b1e62786f8564917 Oct 30 19:30:01.711859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.711815 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerStarted","Data":"f0257c4d166c32e7fc88d3990cd38269aca1f7db094fefbb7b55596a5ac55fab"} Oct 30 19:30:01.711996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.711880 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerStarted","Data":"7a5382cd90ce6066a3e87fee880d4e3f90ea452093890ae42da59e00e533db05"} Oct 30 19:30:01.713616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.713588 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerStarted","Data":"83b39c45425e8ed768f4fb1e7b64d47d69b16f2c2317521c34feec749f413dff"} Oct 30 19:30:01.713736 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.713644 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerStarted","Data":"6216fac07761a1eed87c0145b32fb198efbe46d7fbcd167b7230dbc221425321"} Oct 30 19:30:01.715380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.715351 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerStarted","Data":"3d31e256ffeb2bf92e5cbd2c704f68f19023510e6139fcee6a68bd2da961d5e6"} Oct 30 19:30:01.715479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.715406 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerStarted","Data":"ed2662914a8379469eeabc7311bbe8fd8656f83300633483b1e62786f8564917"} Oct 30 19:30:01.716872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.716849 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerStarted","Data":"6d35979e7f462950f56ad245c73b53ec6acd666e3a0d85a168a55be38c89e834"} Oct 30 19:30:01.716961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.716889 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerStarted","Data":"c4f649f373a2c1ac917ffec160523b0da439b18439ebf2b43b4211c6a8ca91eb"} Oct 30 19:30:01.718526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.718506 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerStarted","Data":"eb6f9d0af25553f7ddffe2631004a8b63b80667c4b4acb7b429a7c912667a0eb"} Oct 30 19:30:01.718607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.718546 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerStarted","Data":"87591a0a1efe8e20bb126dabd095611dab1c406d95963a60eb2522a412317a53"} Oct 30 19:30:01.720147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.720122 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerStarted","Data":"31ae36bdcdae6cbc1936f53dc55d50a04fca22b43104eb71113de4c54344d286"} Oct 30 19:30:01.720276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.720169 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerStarted","Data":"0be5539d014784f559b31297ac6e12c2866016fe05d6782b33a053bc688213a9"} Oct 30 19:30:01.721733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.721715 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerStarted","Data":"16002e5809cbc0b98a2be88ff1fa1c99799922bc8885ce34fecdcadbe957ef7c"} Oct 30 19:30:01.721811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.721756 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerStarted","Data":"5e83837880548ad3a92d3fbc21d7833e9539a5072238842eb971a642ef43d98b"} Oct 30 19:30:01.831131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.829550 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" podStartSLOduration=1.829534224 podStartE2EDuration="1.829534224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:01.827838989 +0000 UTC m=+2638.329749472" watchObservedRunningTime="2026-10-30 19:30:01.829534224 +0000 UTC m=+2638.331444711" Oct 30 19:30:01.938719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:01.938677 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" podStartSLOduration=1.9386645869999999 podStartE2EDuration="1.938664587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:01.937197228 +0000 UTC m=+2638.439107714" watchObservedRunningTime="2026-10-30 19:30:01.938664587 +0000 UTC m=+2638.440575073" Oct 30 19:30:02.046891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.046846 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" podStartSLOduration=2.046834231 podStartE2EDuration="2.046834231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.046157274 +0000 UTC m=+2638.548067770" watchObservedRunningTime="2026-10-30 19:30:02.046834231 +0000 UTC m=+2638.548744714" Oct 30 19:30:02.159549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.159500 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" podStartSLOduration=2.159488568 podStartE2EDuration="2.159488568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.15609941 +0000 UTC m=+2638.658009896" watchObservedRunningTime="2026-10-30 19:30:02.159488568 +0000 UTC m=+2638.661399067" Oct 30 19:30:02.266260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.266167 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" podStartSLOduration=2.266152592 podStartE2EDuration="2.266152592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.265535868 +0000 UTC m=+2638.767446354" watchObservedRunningTime="2026-10-30 19:30:02.266152592 +0000 UTC m=+2638.768063077" Oct 30 19:30:02.382106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.382058 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" podStartSLOduration=2.3820450429999998 podStartE2EDuration="2.382045043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.379930117 +0000 UTC m=+2638.881840603" watchObservedRunningTime="2026-10-30 19:30:02.382045043 +0000 UTC m=+2638.883955528" Oct 30 19:30:02.490597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.490551 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" podStartSLOduration=2.490539397 podStartE2EDuration="2.490539397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.49018768 +0000 UTC m=+2638.992098166" watchObservedRunningTime="2026-10-30 19:30:02.490539397 +0000 UTC m=+2638.992449882" Oct 30 19:30:02.732302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.732266 4789 generic.go:356] "Generic (PLEG): container finished" podID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerID="83b39c45425e8ed768f4fb1e7b64d47d69b16f2c2317521c34feec749f413dff" exitCode=60 Oct 30 19:30:02.732496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.732352 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerDied","Data":"83b39c45425e8ed768f4fb1e7b64d47d69b16f2c2317521c34feec749f413dff"} Oct 30 19:30:02.732573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.732560 4789 scope.go:122] "RemoveContainer" containerID="83b39c45425e8ed768f4fb1e7b64d47d69b16f2c2317521c34feec749f413dff" Oct 30 19:30:02.734187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.734165 4789 generic.go:356] "Generic (PLEG): container finished" podID="17fbeb12-4da0-473b-b064-071f95348b7b" containerID="3d31e256ffeb2bf92e5cbd2c704f68f19023510e6139fcee6a68bd2da961d5e6" exitCode=60 Oct 30 19:30:02.734328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.734238 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerDied","Data":"3d31e256ffeb2bf92e5cbd2c704f68f19023510e6139fcee6a68bd2da961d5e6"} Oct 30 19:30:02.734498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.734481 4789 scope.go:122] "RemoveContainer" containerID="3d31e256ffeb2bf92e5cbd2c704f68f19023510e6139fcee6a68bd2da961d5e6" Oct 30 19:30:02.736430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.736406 4789 generic.go:356] "Generic (PLEG): container finished" podID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerID="6d35979e7f462950f56ad245c73b53ec6acd666e3a0d85a168a55be38c89e834" exitCode=60 Oct 30 19:30:02.736538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.736493 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerDied","Data":"6d35979e7f462950f56ad245c73b53ec6acd666e3a0d85a168a55be38c89e834"} Oct 30 19:30:02.736676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.736657 4789 scope.go:122] "RemoveContainer" containerID="6d35979e7f462950f56ad245c73b53ec6acd666e3a0d85a168a55be38c89e834" Oct 30 19:30:02.742959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.742935 4789 generic.go:356] "Generic (PLEG): container finished" podID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerID="eb6f9d0af25553f7ddffe2631004a8b63b80667c4b4acb7b429a7c912667a0eb" exitCode=60 Oct 30 19:30:02.743142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.743122 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerDied","Data":"eb6f9d0af25553f7ddffe2631004a8b63b80667c4b4acb7b429a7c912667a0eb"} Oct 30 19:30:02.743475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.743271 4789 scope.go:122] "RemoveContainer" containerID="eb6f9d0af25553f7ddffe2631004a8b63b80667c4b4acb7b429a7c912667a0eb" Oct 30 19:30:02.745245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.745192 4789 generic.go:356] "Generic (PLEG): container finished" podID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerID="31ae36bdcdae6cbc1936f53dc55d50a04fca22b43104eb71113de4c54344d286" exitCode=60 Oct 30 19:30:02.745328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.745246 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerDied","Data":"31ae36bdcdae6cbc1936f53dc55d50a04fca22b43104eb71113de4c54344d286"} Oct 30 19:30:02.745511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.745494 4789 scope.go:122] "RemoveContainer" containerID="31ae36bdcdae6cbc1936f53dc55d50a04fca22b43104eb71113de4c54344d286" Oct 30 19:30:02.747375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.747355 4789 generic.go:356] "Generic (PLEG): container finished" podID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerID="16002e5809cbc0b98a2be88ff1fa1c99799922bc8885ce34fecdcadbe957ef7c" exitCode=60 Oct 30 19:30:02.747467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.747451 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerDied","Data":"16002e5809cbc0b98a2be88ff1fa1c99799922bc8885ce34fecdcadbe957ef7c"} Oct 30 19:30:02.747636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.747615 4789 scope.go:122] "RemoveContainer" containerID="16002e5809cbc0b98a2be88ff1fa1c99799922bc8885ce34fecdcadbe957ef7c" Oct 30 19:30:02.749082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.749064 4789 generic.go:356] "Generic (PLEG): container finished" podID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerID="f0257c4d166c32e7fc88d3990cd38269aca1f7db094fefbb7b55596a5ac55fab" exitCode=60 Oct 30 19:30:02.749188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.749117 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerDied","Data":"f0257c4d166c32e7fc88d3990cd38269aca1f7db094fefbb7b55596a5ac55fab"} Oct 30 19:30:02.749360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:02.749345 4789 scope.go:122] "RemoveContainer" containerID="f0257c4d166c32e7fc88d3990cd38269aca1f7db094fefbb7b55596a5ac55fab" Oct 30 19:30:03.757632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.757597 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerStarted","Data":"19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3"} Oct 30 19:30:03.759784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.759756 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerStarted","Data":"5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a"} Oct 30 19:30:03.761714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.761691 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerStarted","Data":"c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db"} Oct 30 19:30:03.763484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.763461 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerStarted","Data":"dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce"} Oct 30 19:30:03.765155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.765133 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerStarted","Data":"d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9"} Oct 30 19:30:03.766801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.766782 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerStarted","Data":"221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765"} Oct 30 19:30:03.768387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:03.768367 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerStarted","Data":"479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b"} Oct 30 19:30:04.776662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.776629 4789 generic.go:356] "Generic (PLEG): container finished" podID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerID="479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b" exitCode=60 Oct 30 19:30:04.777120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.776712 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerDied","Data":"479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b"} Oct 30 19:30:04.777120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.776818 4789 scope.go:122] "RemoveContainer" containerID="f0257c4d166c32e7fc88d3990cd38269aca1f7db094fefbb7b55596a5ac55fab" Oct 30 19:30:04.777120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.776881 4789 scope.go:122] "RemoveContainer" containerID="479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b" Oct 30 19:30:04.777120 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.777089 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff_odh-ai-gateway-infra(0b80746e-ded7-431a-b8f1-3e4f1713f31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" Oct 30 19:30:04.778792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.778766 4789 generic.go:356] "Generic (PLEG): container finished" podID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerID="19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3" exitCode=60 Oct 30 19:30:04.778886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.778842 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerDied","Data":"19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3"} Oct 30 19:30:04.779015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.779002 4789 scope.go:122] "RemoveContainer" containerID="19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3" Oct 30 19:30:04.779219 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.779189 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr_odh-ai-gateway-infra(5531d96f-29f5-488c-bf3a-b3f1ef5a719a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" Oct 30 19:30:04.780622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.780604 4789 generic.go:356] "Generic (PLEG): container finished" podID="17fbeb12-4da0-473b-b064-071f95348b7b" containerID="5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a" exitCode=60 Oct 30 19:30:04.780714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.780670 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerDied","Data":"5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a"} Oct 30 19:30:04.780827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.780813 4789 scope.go:122] "RemoveContainer" containerID="5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a" Oct 30 19:30:04.781013 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.780994 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2_odh-ai-gateway-infra(17fbeb12-4da0-473b-b064-071f95348b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" Oct 30 19:30:04.782460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.782434 4789 generic.go:356] "Generic (PLEG): container finished" podID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerID="c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db" exitCode=60 Oct 30 19:30:04.782586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.782520 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerDied","Data":"c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db"} Oct 30 19:30:04.782698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.782679 4789 scope.go:122] "RemoveContainer" containerID="c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db" Oct 30 19:30:04.782904 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.782884 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p_odh-ai-gateway-infra(2de2cfb0-4f3c-4684-b68f-8e87b985bd7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" Oct 30 19:30:04.784415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.784391 4789 generic.go:356] "Generic (PLEG): container finished" podID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerID="dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce" exitCode=60 Oct 30 19:30:04.784493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.784466 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerDied","Data":"dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce"} Oct 30 19:30:04.784638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.784625 4789 scope.go:122] "RemoveContainer" containerID="dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce" Oct 30 19:30:04.784811 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.784790 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q_odh-ai-gateway-infra(eee35881-5f88-4564-bc41-a50e1fbf40da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" Oct 30 19:30:04.786351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.786329 4789 generic.go:356] "Generic (PLEG): container finished" podID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerID="d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9" exitCode=60 Oct 30 19:30:04.786432 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.786406 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerDied","Data":"d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9"} Oct 30 19:30:04.786580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.786563 4789 scope.go:122] "RemoveContainer" containerID="d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9" Oct 30 19:30:04.786789 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.786771 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6_odh-ai-gateway-infra(15b9e5cd-a841-42c8-a898-3c0206a42ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" Oct 30 19:30:04.788253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.788234 4789 generic.go:356] "Generic (PLEG): container finished" podID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerID="221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765" exitCode=60 Oct 30 19:30:04.788335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.788268 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerDied","Data":"221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765"} Oct 30 19:30:04.788490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.788473 4789 scope.go:122] "RemoveContainer" containerID="221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765" Oct 30 19:30:04.788650 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:04.788634 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64_odh-ai-gateway-infra(19b3ee7d-f79a-4a99-bc48-5ea894290b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" Oct 30 19:30:04.790516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.790498 4789 scope.go:122] "RemoveContainer" containerID="83b39c45425e8ed768f4fb1e7b64d47d69b16f2c2317521c34feec749f413dff" Oct 30 19:30:04.804835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.804817 4789 scope.go:122] "RemoveContainer" containerID="3d31e256ffeb2bf92e5cbd2c704f68f19023510e6139fcee6a68bd2da961d5e6" Oct 30 19:30:04.818716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.818697 4789 scope.go:122] "RemoveContainer" containerID="6d35979e7f462950f56ad245c73b53ec6acd666e3a0d85a168a55be38c89e834" Oct 30 19:30:04.831875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.831855 4789 scope.go:122] "RemoveContainer" containerID="eb6f9d0af25553f7ddffe2631004a8b63b80667c4b4acb7b429a7c912667a0eb" Oct 30 19:30:04.844291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.844273 4789 scope.go:122] "RemoveContainer" containerID="31ae36bdcdae6cbc1936f53dc55d50a04fca22b43104eb71113de4c54344d286" Oct 30 19:30:04.857621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:04.857608 4789 scope.go:122] "RemoveContainer" containerID="16002e5809cbc0b98a2be88ff1fa1c99799922bc8885ce34fecdcadbe957ef7c" Oct 30 19:30:05.796616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.796584 4789 scope.go:122] "RemoveContainer" containerID="221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765" Oct 30 19:30:05.797093 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.796877 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64_odh-ai-gateway-infra(19b3ee7d-f79a-4a99-bc48-5ea894290b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" Oct 30 19:30:05.798335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.798311 4789 scope.go:122] "RemoveContainer" containerID="479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b" Oct 30 19:30:05.798598 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.798574 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff_odh-ai-gateway-infra(0b80746e-ded7-431a-b8f1-3e4f1713f31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" Oct 30 19:30:05.799992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.799972 4789 scope.go:122] "RemoveContainer" containerID="19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3" Oct 30 19:30:05.800233 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.800194 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr_odh-ai-gateway-infra(5531d96f-29f5-488c-bf3a-b3f1ef5a719a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" Oct 30 19:30:05.801606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.801586 4789 scope.go:122] "RemoveContainer" containerID="5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a" Oct 30 19:30:05.801820 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.801795 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2_odh-ai-gateway-infra(17fbeb12-4da0-473b-b064-071f95348b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" Oct 30 19:30:05.803136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.803115 4789 scope.go:122] "RemoveContainer" containerID="c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db" Oct 30 19:30:05.803403 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.803370 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p_odh-ai-gateway-infra(2de2cfb0-4f3c-4684-b68f-8e87b985bd7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" Oct 30 19:30:05.804986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.804964 4789 scope.go:122] "RemoveContainer" containerID="dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce" Oct 30 19:30:05.805181 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.805162 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q_odh-ai-gateway-infra(eee35881-5f88-4564-bc41-a50e1fbf40da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" Oct 30 19:30:05.806473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:05.806457 4789 scope.go:122] "RemoveContainer" containerID="d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9" Oct 30 19:30:05.806613 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:05.806597 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6_odh-ai-gateway-infra(15b9e5cd-a841-42c8-a898-3c0206a42ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" Oct 30 19:30:06.085486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:06.085403 4789 scope.go:122] "RemoveContainer" containerID="3257e49fd12449b0b652c56d6928b87ae527b9ab150fa7bd6ad06abd0290888a" Oct 30 19:30:06.096174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:06.096147 4789 scope.go:122] "RemoveContainer" containerID="84da86575171ab170bc63fec5ecdcb489a7da17de16d17b0e0346a850d2da913" Oct 30 19:30:14.661119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:14.661050 4789 scope.go:122] "RemoveContainer" containerID="19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3" Oct 30 19:30:14.661535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:14.661198 4789 scope.go:122] "RemoveContainer" containerID="d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9" Oct 30 19:30:15.661584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.661551 4789 scope.go:122] "RemoveContainer" containerID="c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db" Oct 30 19:30:15.874842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.874791 4789 generic.go:356] "Generic (PLEG): container finished" podID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerID="10d17483911bc7d5bba5c1b4e8dd2ae18fbc1ed40d066b3b97476b739245f9a5" exitCode=60 Oct 30 19:30:15.874931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.874864 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerDied","Data":"10d17483911bc7d5bba5c1b4e8dd2ae18fbc1ed40d066b3b97476b739245f9a5"} Oct 30 19:30:15.874931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.874925 4789 scope.go:122] "RemoveContainer" containerID="d38a1a59edc57c4968dbce799d3d0036e5e6588ca71d4b653960d31a74cbf8d9" Oct 30 19:30:15.875236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.875186 4789 scope.go:122] "RemoveContainer" containerID="10d17483911bc7d5bba5c1b4e8dd2ae18fbc1ed40d066b3b97476b739245f9a5" Oct 30 19:30:15.875493 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:15.875458 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6_odh-ai-gateway-infra(15b9e5cd-a841-42c8-a898-3c0206a42ed1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" Oct 30 19:30:15.877012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.876995 4789 generic.go:356] "Generic (PLEG): container finished" podID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerID="bd185bfac4a275684e28fb1ad62f9e647efd41b8b345b619f1703eb76ae032c3" exitCode=60 Oct 30 19:30:15.877080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.877030 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerDied","Data":"bd185bfac4a275684e28fb1ad62f9e647efd41b8b345b619f1703eb76ae032c3"} Oct 30 19:30:15.877315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.877300 4789 scope.go:122] "RemoveContainer" containerID="bd185bfac4a275684e28fb1ad62f9e647efd41b8b345b619f1703eb76ae032c3" Oct 30 19:30:15.877485 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:15.877468 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr_odh-ai-gateway-infra(5531d96f-29f5-488c-bf3a-b3f1ef5a719a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" Oct 30 19:30:15.888282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:15.888263 4789 scope.go:122] "RemoveContainer" containerID="19a94c45e4bbfc9ae34faad94a07cac98b8d3b2a602afc8bccfe2467b2c5a3c3" Oct 30 19:30:16.661638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.661561 4789 scope.go:122] "RemoveContainer" containerID="221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765" Oct 30 19:30:16.889521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.889487 4789 generic.go:356] "Generic (PLEG): container finished" podID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerID="66bf58199d618ca86436bf9f28d6ea4a4c54f738e06692aae638bd8741808c67" exitCode=60 Oct 30 19:30:16.889697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.889559 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerDied","Data":"66bf58199d618ca86436bf9f28d6ea4a4c54f738e06692aae638bd8741808c67"} Oct 30 19:30:16.889697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.889622 4789 scope.go:122] "RemoveContainer" containerID="c0756e0306d283b0d236c7c0f1349174412ef8ba6c2a659bdbb5ffd0784d65db" Oct 30 19:30:16.890095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.889991 4789 scope.go:122] "RemoveContainer" containerID="66bf58199d618ca86436bf9f28d6ea4a4c54f738e06692aae638bd8741808c67" Oct 30 19:30:16.890367 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:16.890342 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p_odh-ai-gateway-infra(2de2cfb0-4f3c-4684-b68f-8e87b985bd7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" Oct 30 19:30:16.996621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:16.996596 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:17.111549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.111505 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:17.157261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.156684 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:17.264454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.264424 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:17.276838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.276816 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\") pod \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " Oct 30 19:30:17.277026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.277010 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\" (UniqueName: \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\") pod \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\" (UID: \"15b9e5cd-a841-42c8-a898-3c0206a42ed1\") " Oct 30 19:30:17.277265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.277192 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca" pod "15b9e5cd-a841-42c8-a898-3c0206a42ed1" (UID: "15b9e5cd-a841-42c8-a898-3c0206a42ed1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:17.277497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.277479 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b9e5cd-a841-42c8-a898-3c0206a42ed1-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:17.278932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.278905 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7" pod "15b9e5cd-a841-42c8-a898-3c0206a42ed1" (UID: "15b9e5cd-a841-42c8-a898-3c0206a42ed1"). InnerVolumeSpecName "kube-api-access-c65w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:17.378380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.378341 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\" (UniqueName: \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\") pod \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " Oct 30 19:30:17.378568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.378441 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\") pod \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\" (UID: \"5531d96f-29f5-488c-bf3a-b3f1ef5a719a\") " Oct 30 19:30:17.378946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.378915 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca" pod "5531d96f-29f5-488c-bf3a-b3f1ef5a719a" (UID: "5531d96f-29f5-488c-bf3a-b3f1ef5a719a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:17.378946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.378935 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65w7\" (UniqueName: \"kubernetes.io/projected/15b9e5cd-a841-42c8-a898-3c0206a42ed1-kube-api-access-c65w7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:17.380904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.380865 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl" pod "5531d96f-29f5-488c-bf3a-b3f1ef5a719a" (UID: "5531d96f-29f5-488c-bf3a-b3f1ef5a719a"). InnerVolumeSpecName "kube-api-access-rzdnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:17.479776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.479737 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdnl\" (UniqueName: \"kubernetes.io/projected/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-kube-api-access-rzdnl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:17.479776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.479770 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5531d96f-29f5-488c-bf3a-b3f1ef5a719a-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:17.661077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.661035 4789 scope.go:122] "RemoveContainer" containerID="dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce" Oct 30 19:30:17.661320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.661169 4789 scope.go:122] "RemoveContainer" containerID="479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b" Oct 30 19:30:17.901738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.901669 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" Oct 30 19:30:17.901738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.901685 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr" event={"ID":"5531d96f-29f5-488c-bf3a-b3f1ef5a719a","Type":"ContainerDied","Data":"6216fac07761a1eed87c0145b32fb198efbe46d7fbcd167b7230dbc221425321"} Oct 30 19:30:17.902235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.901752 4789 scope.go:122] "RemoveContainer" containerID="bd185bfac4a275684e28fb1ad62f9e647efd41b8b345b619f1703eb76ae032c3" Oct 30 19:30:17.905699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.905676 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" event={"ID":"15b9e5cd-a841-42c8-a898-3c0206a42ed1","Type":"ContainerDied","Data":"0be5539d014784f559b31297ac6e12c2866016fe05d6782b33a053bc688213a9"} Oct 30 19:30:17.905803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.905744 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6" Oct 30 19:30:17.907739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.907715 4789 generic.go:356] "Generic (PLEG): container finished" podID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerID="222c81345d2bd437e03fd2ed9d50c0d781a312843d4b06e5ecb5b7d59cfaf2a1" exitCode=60 Oct 30 19:30:17.907830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.907764 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerDied","Data":"222c81345d2bd437e03fd2ed9d50c0d781a312843d4b06e5ecb5b7d59cfaf2a1"} Oct 30 19:30:17.908123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.908108 4789 scope.go:122] "RemoveContainer" containerID="222c81345d2bd437e03fd2ed9d50c0d781a312843d4b06e5ecb5b7d59cfaf2a1" Oct 30 19:30:17.908376 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:17.908352 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64_odh-ai-gateway-infra(19b3ee7d-f79a-4a99-bc48-5ea894290b36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" Oct 30 19:30:17.915987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.915971 4789 scope.go:122] "RemoveContainer" containerID="10d17483911bc7d5bba5c1b4e8dd2ae18fbc1ed40d066b3b97476b739245f9a5" Oct 30 19:30:17.924356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:17.924341 4789 scope.go:122] "RemoveContainer" containerID="221c585578eff727c6806ad4531155935f8f8fa0c24918a5c594bb1932d0b765" Oct 30 19:30:18.010288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.010269 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:18.169618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.169592 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:18.172323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.172300 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889810-znxgr"] Oct 30 19:30:18.179600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.179579 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:18.197138 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:18.197102 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:30:18.275154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.275126 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:18.280723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.280702 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889810-7nsx6"] Oct 30 19:30:18.289819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.289798 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\") pod \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " Oct 30 19:30:18.289902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.289830 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\" (UniqueName: \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\") pod \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\" (UID: \"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a\") " Oct 30 19:30:18.290133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.290113 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca" pod "2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" (UID: "2de2cfb0-4f3c-4684-b68f-8e87b985bd7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:18.290247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.290229 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:18.291852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.291833 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp" pod "2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" (UID: "2de2cfb0-4f3c-4684-b68f-8e87b985bd7a"). InnerVolumeSpecName "kube-api-access-8slnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:18.391273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.391243 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8slnp\" (UniqueName: \"kubernetes.io/projected/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a-kube-api-access-8slnp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:18.661012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.660980 4789 scope.go:122] "RemoveContainer" containerID="5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a" Oct 30 19:30:18.917772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.917750 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" Oct 30 19:30:18.918129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.917751 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p" event={"ID":"2de2cfb0-4f3c-4684-b68f-8e87b985bd7a","Type":"ContainerDied","Data":"c4f649f373a2c1ac917ffec160523b0da439b18439ebf2b43b4211c6a8ca91eb"} Oct 30 19:30:18.918129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.917877 4789 scope.go:122] "RemoveContainer" containerID="66bf58199d618ca86436bf9f28d6ea4a4c54f738e06692aae638bd8741808c67" Oct 30 19:30:18.919747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.919725 4789 generic.go:356] "Generic (PLEG): container finished" podID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerID="8ec6a88dc1da8dddf620a18cfe321094880c7a3020f6513a135e13c404b5341a" exitCode=60 Oct 30 19:30:18.919857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.919792 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerDied","Data":"8ec6a88dc1da8dddf620a18cfe321094880c7a3020f6513a135e13c404b5341a"} Oct 30 19:30:18.920175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.920147 4789 scope.go:122] "RemoveContainer" containerID="8ec6a88dc1da8dddf620a18cfe321094880c7a3020f6513a135e13c404b5341a" Oct 30 19:30:18.920423 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:18.920404 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q_odh-ai-gateway-infra(eee35881-5f88-4564-bc41-a50e1fbf40da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" Oct 30 19:30:18.924754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.924730 4789 generic.go:356] "Generic (PLEG): container finished" podID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerID="27464b3c03b79c2be8081c2113fd19c235fb28ec99dfb11ab64bbae244ed3810" exitCode=60 Oct 30 19:30:18.924842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.924798 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerDied","Data":"27464b3c03b79c2be8081c2113fd19c235fb28ec99dfb11ab64bbae244ed3810"} Oct 30 19:30:18.925116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.925100 4789 scope.go:122] "RemoveContainer" containerID="27464b3c03b79c2be8081c2113fd19c235fb28ec99dfb11ab64bbae244ed3810" Oct 30 19:30:18.925368 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:18.925348 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff_odh-ai-gateway-infra(0b80746e-ded7-431a-b8f1-3e4f1713f31e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" Oct 30 19:30:18.931036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.931009 4789 scope.go:122] "RemoveContainer" containerID="dd99fc1778726c7396721c5c2b74e6c47b3e7e6ed0b1bd2ebdc889e6c54093ce" Oct 30 19:30:18.943304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:18.943288 4789 scope.go:122] "RemoveContainer" containerID="479ff3717d749a8fef3020f1d6a7cd54a1ce2c5a2e2b680fd9a46d36c21dec3b" Oct 30 19:30:19.071975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.071954 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:19.217266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.217245 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:19.304708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.304673 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:19.305463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.305433 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\" (UniqueName: \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\") pod \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " Oct 30 19:30:19.305588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.305482 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\") pod \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\" (UID: \"19b3ee7d-f79a-4a99-bc48-5ea894290b36\") " Oct 30 19:30:19.306052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.306028 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca" pod "19b3ee7d-f79a-4a99-bc48-5ea894290b36" (UID: "19b3ee7d-f79a-4a99-bc48-5ea894290b36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:19.306229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.306183 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b3ee7d-f79a-4a99-bc48-5ea894290b36-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.307330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.307309 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4833f3-29889810-6vp7p"] Oct 30 19:30:19.307819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.307798 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6" pod "19b3ee7d-f79a-4a99-bc48-5ea894290b36" (UID: "19b3ee7d-f79a-4a99-bc48-5ea894290b36"). InnerVolumeSpecName "kube-api-access-5rmm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:19.407357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.407336 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rmm6\" (UniqueName: \"kubernetes.io/projected/19b3ee7d-f79a-4a99-bc48-5ea894290b36-kube-api-access-5rmm6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:19.669981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.669947 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" path="/var/lib/kubelet/pods/15b9e5cd-a841-42c8-a898-3c0206a42ed1/volumes" Oct 30 19:30:19.670487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.670464 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" path="/var/lib/kubelet/pods/2de2cfb0-4f3c-4684-b68f-8e87b985bd7a/volumes" Oct 30 19:30:19.670907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.670888 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" path="/var/lib/kubelet/pods/5531d96f-29f5-488c-bf3a-b3f1ef5a719a/volumes" Oct 30 19:30:19.932325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.932201 4789 generic.go:356] "Generic (PLEG): container finished" podID="17fbeb12-4da0-473b-b064-071f95348b7b" containerID="181f33f41539a37c76ce98179afed7916d5feb9372148e83da703f020f0d46aa" exitCode=60 Oct 30 19:30:19.932325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.932237 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerDied","Data":"181f33f41539a37c76ce98179afed7916d5feb9372148e83da703f020f0d46aa"} Oct 30 19:30:19.932325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.932310 4789 scope.go:122] "RemoveContainer" containerID="5b5705ee28f876d5329c8e9dcd836d33c32d7282145b25dcf930b166d2280d0a" Oct 30 19:30:19.932896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.932701 4789 scope.go:122] "RemoveContainer" containerID="181f33f41539a37c76ce98179afed7916d5feb9372148e83da703f020f0d46aa" Oct 30 19:30:19.932986 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:19.932957 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2_odh-ai-gateway-infra(17fbeb12-4da0-473b-b064-071f95348b7b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" Oct 30 19:30:19.936512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.936495 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" Oct 30 19:30:19.936610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.936519 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64" event={"ID":"19b3ee7d-f79a-4a99-bc48-5ea894290b36","Type":"ContainerDied","Data":"5e83837880548ad3a92d3fbc21d7833e9539a5072238842eb971a642ef43d98b"} Oct 30 19:30:19.946851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:19.946822 4789 scope.go:122] "RemoveContainer" containerID="222c81345d2bd437e03fd2ed9d50c0d781a312843d4b06e5ecb5b7d59cfaf2a1" Oct 30 19:30:20.094885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.094850 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:20.157523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.157494 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:20.160728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.160701 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-1323a7-29889810-nlk64"] Oct 30 19:30:20.215018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.214992 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:20.263743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.263716 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:20.374741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.374716 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:20.417746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.417720 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\" (UniqueName: \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\") pod \"eee35881-5f88-4564-bc41-a50e1fbf40da\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " Oct 30 19:30:20.417914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.417816 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\") pod \"eee35881-5f88-4564-bc41-a50e1fbf40da\" (UID: \"eee35881-5f88-4564-bc41-a50e1fbf40da\") " Oct 30 19:30:20.418203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.418183 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca" pod "eee35881-5f88-4564-bc41-a50e1fbf40da" (UID: "eee35881-5f88-4564-bc41-a50e1fbf40da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.419675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.419652 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6" pod "eee35881-5f88-4564-bc41-a50e1fbf40da" (UID: "eee35881-5f88-4564-bc41-a50e1fbf40da"). InnerVolumeSpecName "kube-api-access-r82k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.519119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.519051 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\" (UniqueName: \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\") pod \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " Oct 30 19:30:20.519285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.519162 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\") pod \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\" (UID: \"0b80746e-ded7-431a-b8f1-3e4f1713f31e\") " Oct 30 19:30:20.519475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.519457 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82k6\" (UniqueName: \"kubernetes.io/projected/eee35881-5f88-4564-bc41-a50e1fbf40da-kube-api-access-r82k6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.519475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.519477 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee35881-5f88-4564-bc41-a50e1fbf40da-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.519639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.519520 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca" pod "0b80746e-ded7-431a-b8f1-3e4f1713f31e" (UID: "0b80746e-ded7-431a-b8f1-3e4f1713f31e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.521201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.521177 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48" pod "0b80746e-ded7-431a-b8f1-3e4f1713f31e" (UID: "0b80746e-ded7-431a-b8f1-3e4f1713f31e"). InnerVolumeSpecName "kube-api-access-8tq48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.620761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.620728 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tq48\" (UniqueName: \"kubernetes.io/projected/0b80746e-ded7-431a-b8f1-3e4f1713f31e-kube-api-access-8tq48\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.620761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.620753 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b80746e-ded7-431a-b8f1-3e4f1713f31e-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.945682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.945657 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" Oct 30 19:30:20.946158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.945690 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q" event={"ID":"eee35881-5f88-4564-bc41-a50e1fbf40da","Type":"ContainerDied","Data":"87591a0a1efe8e20bb126dabd095611dab1c406d95963a60eb2522a412317a53"} Oct 30 19:30:20.946158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.945761 4789 scope.go:122] "RemoveContainer" containerID="8ec6a88dc1da8dddf620a18cfe321094880c7a3020f6513a135e13c404b5341a" Oct 30 19:30:20.948353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.948256 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" Oct 30 19:30:20.948353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.948278 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff" event={"ID":"0b80746e-ded7-431a-b8f1-3e4f1713f31e","Type":"ContainerDied","Data":"7a5382cd90ce6066a3e87fee880d4e3f90ea452093890ae42da59e00e533db05"} Oct 30 19:30:20.958204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:20.958175 4789 scope.go:122] "RemoveContainer" containerID="27464b3c03b79c2be8081c2113fd19c235fb28ec99dfb11ab64bbae244ed3810" Oct 30 19:30:21.059124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.059103 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:21.102172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.102135 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:21.108627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.108600 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889810-dtnff"] Oct 30 19:30:21.205076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.205024 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:21.211279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.211255 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:21.215348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.215327 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a790b-29889810-bmz5q"] Oct 30 19:30:21.329698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.329665 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\" (UniqueName: \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\") pod \"17fbeb12-4da0-473b-b064-071f95348b7b\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " Oct 30 19:30:21.329875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.329856 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\") pod \"17fbeb12-4da0-473b-b064-071f95348b7b\" (UID: \"17fbeb12-4da0-473b-b064-071f95348b7b\") " Oct 30 19:30:21.330297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.330265 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca" pod "17fbeb12-4da0-473b-b064-071f95348b7b" (UID: "17fbeb12-4da0-473b-b064-071f95348b7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:21.331895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.331862 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4" pod "17fbeb12-4da0-473b-b064-071f95348b7b" (UID: "17fbeb12-4da0-473b-b064-071f95348b7b"). InnerVolumeSpecName "kube-api-access-pmch4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.431506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.431470 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fbeb12-4da0-473b-b064-071f95348b7b-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.431506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.431507 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmch4\" (UniqueName: \"kubernetes.io/projected/17fbeb12-4da0-473b-b064-071f95348b7b-kube-api-access-pmch4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.667742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.667671 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" path="/var/lib/kubelet/pods/0b80746e-ded7-431a-b8f1-3e4f1713f31e/volumes" Oct 30 19:30:21.668115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.668099 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" path="/var/lib/kubelet/pods/19b3ee7d-f79a-4a99-bc48-5ea894290b36/volumes" Oct 30 19:30:21.668527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.668513 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" path="/var/lib/kubelet/pods/eee35881-5f88-4564-bc41-a50e1fbf40da/volumes" Oct 30 19:30:21.958539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.958514 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" Oct 30 19:30:21.959024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.958514 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2" event={"ID":"17fbeb12-4da0-473b-b064-071f95348b7b","Type":"ContainerDied","Data":"ed2662914a8379469eeabc7311bbe8fd8656f83300633483b1e62786f8564917"} Oct 30 19:30:21.959024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:21.958627 4789 scope.go:122] "RemoveContainer" containerID="181f33f41539a37c76ce98179afed7916d5feb9372148e83da703f020f0d46aa" Oct 30 19:30:22.144969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:22.144940 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:22.190744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:22.190720 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889810-rpzl2"] Oct 30 19:30:23.668811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:23.668780 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" path="/var/lib/kubelet/pods/17fbeb12-4da0-473b-b064-071f95348b7b/volumes" Oct 30 19:30:26.348516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.348485 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56"] Oct 30 19:30:26.349670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.349646 4789 cpu_manager.go:414] "containerMap: removing container" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.349818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.349806 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.349907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.349896 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.349975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.349965 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.350046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350038 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.350110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350101 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.350179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350171 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.350264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350253 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.350412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350401 4789 cpu_manager.go:414] "containerMap: removing container" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.350939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.350921 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.351039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351029 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.351110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351101 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.351194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351185 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.351291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351279 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.351365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351356 4789 cpu_manager.go:414] "containerMap: removing container" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.351449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351428 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.351533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351519 4789 cpu_manager.go:414] "containerMap: removing container" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.351604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351594 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.351676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351667 4789 cpu_manager.go:414] "containerMap: removing container" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.351742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351733 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.351811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351803 4789 cpu_manager.go:414] "containerMap: removing container" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.351881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351871 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.351958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.351950 4789 cpu_manager.go:414] "containerMap: removing container" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.352027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352018 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.352092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352083 4789 cpu_manager.go:414] "containerMap: removing container" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.352162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352152 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.352250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352238 4789 cpu_manager.go:414] "containerMap: removing container" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.352323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352313 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.352427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352416 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.352501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352492 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.352597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352587 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.352678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352668 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.352895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352883 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.352978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.352969 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.353047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353039 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.353117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353109 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:26.353187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353178 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.353298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353288 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.353369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353361 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:26.353836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353817 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.353984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.353972 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:26.354064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354055 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.354138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354130 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:26.354235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354202 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.354316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354307 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.354392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354383 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.354466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354458 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:26.354540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354531 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:26.354609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.354601 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:26.360040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.360022 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.408419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.408386 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56"] Oct 30 19:30:26.411271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.411250 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:26.449712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.449688 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:30:26.449826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.449713 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:30:26.449826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.449792 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:30:26.483853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.483832 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.483963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.483950 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.484016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.484005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krchb\" (UniqueName: \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.585408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.585387 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.585506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.585432 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krchb\" (UniqueName: \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.585506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.585479 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.586129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.586107 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.587748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.587724 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.642886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.642827 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krchb\" (UniqueName: \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\") pod \"maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:26.677755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:26.677733 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:27.056735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:27.056706 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56"] Oct 30 19:30:27.059291 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:27.059263 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c81b220_9f81_41ba_b0ea_53ee0b873997.slice/crio-0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0 WatchSource:0}: Error finding container 0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0: Status 404 returned error can't find the container with id 0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0 Oct 30 19:30:27.998200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:27.998171 4789 generic.go:356] "Generic (PLEG): container finished" podID="5c81b220-9f81-41ba-b0ea-53ee0b873997" containerID="34fd03566ac7e15b6f3d458df634788d2a5721e4fd73e43ed47e56d17d3a9183" exitCode=0 Oct 30 19:30:27.998499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:27.998236 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" event={"ID":"5c81b220-9f81-41ba-b0ea-53ee0b873997","Type":"ContainerDied","Data":"34fd03566ac7e15b6f3d458df634788d2a5721e4fd73e43ed47e56d17d3a9183"} Oct 30 19:30:27.998499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:27.998282 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" event={"ID":"5c81b220-9f81-41ba-b0ea-53ee0b873997","Type":"ContainerStarted","Data":"0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0"} Oct 30 19:30:29.153391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.153368 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:29.310597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.310520 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\" (UniqueName: \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\") pod \"5c81b220-9f81-41ba-b0ea-53ee0b873997\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " Oct 30 19:30:29.310597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.310572 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\") pod \"5c81b220-9f81-41ba-b0ea-53ee0b873997\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " Oct 30 19:30:29.310806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.310669 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\") pod \"5c81b220-9f81-41ba-b0ea-53ee0b873997\" (UID: \"5c81b220-9f81-41ba-b0ea-53ee0b873997\") " Oct 30 19:30:29.311071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.311041 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca" pod "5c81b220-9f81-41ba-b0ea-53ee0b873997" (UID: "5c81b220-9f81-41ba-b0ea-53ee0b873997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:29.312698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.312674 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca" pod "5c81b220-9f81-41ba-b0ea-53ee0b873997" (UID: "5c81b220-9f81-41ba-b0ea-53ee0b873997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:29.313032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.313016 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb" pod "5c81b220-9f81-41ba-b0ea-53ee0b873997" (UID: "5c81b220-9f81-41ba-b0ea-53ee0b873997"). InnerVolumeSpecName "kube-api-access-krchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:29.412455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.412435 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:29.412455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.412455 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c81b220-9f81-41ba-b0ea-53ee0b873997-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:29.412578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:29.412466 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krchb\" (UniqueName: \"kubernetes.io/projected/5c81b220-9f81-41ba-b0ea-53ee0b873997-kube-api-access-krchb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:30.011461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:30.011430 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" Oct 30 19:30:30.011626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:30.011472 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56" event={"ID":"5c81b220-9f81-41ba-b0ea-53ee0b873997","Type":"ContainerDied","Data":"0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0"} Oct 30 19:30:30.011626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:30.011518 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5a8947925ff0f7e3773489c867747eb9c5672efe413827617af53344b39af0" Oct 30 19:30:37.074928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:37.074894 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:30:37.075413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:37.075163 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" containerID="cri-o://3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c" gracePeriod=30 Oct 30 19:30:37.536978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:37.536944 4789 patch_prober.go:29] interesting pod/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:30:37.537145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:37.537011 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:30:37.761634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:37.761584 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:30:41.031822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:41.031782 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:30:41.032427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:41.032025 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" containerID="cri-o://88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f" gracePeriod=30 Oct 30 19:30:42.332177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.332152 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:30:42.438377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438344 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438434 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438465 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438495 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438543 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438573 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438605 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438637 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438703 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\") pod \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\" (UID: \"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf\") " Oct 30 19:30:42.438991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.438903 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:42.439429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439117 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:42.439429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439244 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.439429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439261 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.439429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439366 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:42.439724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439505 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:42.439724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.439546 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:42.440756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.440730 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:30:42.440935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.440919 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:42.441098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.441077 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:42.441159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.441108 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd" pod "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" (UID: "e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf"). InnerVolumeSpecName "kube-api-access-6lpvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:42.540014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.539980 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540007 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540016 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540025 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpvd\" (UniqueName: \"kubernetes.io/projected/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-kube-api-access-6lpvd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540036 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540044 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.540329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.540052 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:42.762481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.762394 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" podUID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" containerName="payload-processing" containerID="cri-o://471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51" gracePeriod=25 Oct 30 19:30:42.998927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:42.998903 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:30:43.044047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.043980 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\" (UniqueName: \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\") pod \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " Oct 30 19:30:43.044222 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.044195 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\") pod \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\" (UID: \"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3\") " Oct 30 19:30:43.044522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.044496 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume" pod "e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" (UID: "e042745e-ef0d-4f87-9b6c-fffa8c3d81b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:43.044640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.044609 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:43.045968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.045950 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l" pod "e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" (UID: "e042745e-ef0d-4f87-9b6c-fffa8c3d81b3"). InnerVolumeSpecName "kube-api-access-vbh5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:43.113687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.113499 4789 generic.go:356] "Generic (PLEG): container finished" podID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" containerID="471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51" exitCode=0 Oct 30 19:30:43.113687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.113553 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" Oct 30 19:30:43.113687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.113590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" event={"ID":"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3","Type":"ContainerDied","Data":"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51"} Oct 30 19:30:43.113687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.113655 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd" event={"ID":"e042745e-ef0d-4f87-9b6c-fffa8c3d81b3","Type":"ContainerDied","Data":"e6270e44d2d17dd1af7f4820b483de890c722714a10803baac80ca0bf3d07537"} Oct 30 19:30:43.113687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.113681 4789 scope.go:122] "RemoveContainer" containerID="471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51" Oct 30 19:30:43.115638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.115613 4789 generic.go:356] "Generic (PLEG): container finished" podID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerID="3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c" exitCode=0 Oct 30 19:30:43.115756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.115649 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" Oct 30 19:30:43.115756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.115724 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" event={"ID":"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf","Type":"ContainerDied","Data":"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c"} Oct 30 19:30:43.115886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.115764 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp" event={"ID":"e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf","Type":"ContainerDied","Data":"e10ad03b30e11279e683f9e9d9dc3abd3f140eff12d13a315b201ba9c261f332"} Oct 30 19:30:43.126871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.126502 4789 scope.go:122] "RemoveContainer" containerID="471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51" Oct 30 19:30:43.126989 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:43.126915 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51\": container with ID starting with 471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51 not found: ID does not exist" containerID="471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51" Oct 30 19:30:43.126989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.126947 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51"} err="failed to get container status \"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51\": rpc error: code = NotFound desc = could not find container \"471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51\": container with ID starting with 471c20a516acd1cbbf8d867d14a91db05cac9cd941fa236589f6e1e967d9cb51 not found: ID does not exist" Oct 30 19:30:43.126989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.126974 4789 scope.go:122] "RemoveContainer" containerID="3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c" Oct 30 19:30:43.139098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.139076 4789 scope.go:122] "RemoveContainer" containerID="3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c" Oct 30 19:30:43.139804 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:43.139782 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c\": container with ID starting with 3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c not found: ID does not exist" containerID="3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c" Oct 30 19:30:43.139916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.139812 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c"} err="failed to get container status \"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c\": rpc error: code = NotFound desc = could not find container \"3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c\": container with ID starting with 3f4055a769471d6b3cc21b0c28ed4805611cfe6f7108a4bbe4601c71b1b1ef3c not found: ID does not exist" Oct 30 19:30:43.145440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.145418 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbh5l\" (UniqueName: \"kubernetes.io/projected/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3-kube-api-access-vbh5l\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:43.272570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.272542 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:30:43.276880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.276854 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-1323a7-openshift-default-7b64d489ff-4fjxp"] Oct 30 19:30:43.380094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.380033 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:30:43.384888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.384862 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4833f3-ffc4ffc54-grxtd"] Oct 30 19:30:43.667865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.667789 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" path="/var/lib/kubelet/pods/e042745e-ef0d-4f87-9b6c-fffa8c3d81b3/volumes" Oct 30 19:30:43.668256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:43.668240 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" path="/var/lib/kubelet/pods/e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf/volumes" Oct 30 19:30:45.743098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743059 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743510 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743522 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80746e-ded7-431a-b8f1-3e4f1713f31e" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743540 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5c81b220-9f81-41ba-b0ea-53ee0b873997" containerName="revoke-keys" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743545 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c81b220-9f81-41ba-b0ea-53ee0b873997" containerName="revoke-keys" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743558 4789 cpu_manager.go:414] "containerMap: removing container" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743563 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743569 4789 cpu_manager.go:414] "containerMap: removing container" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743574 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743580 4789 cpu_manager.go:414] "containerMap: removing container" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743586 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b3ee7d-f79a-4a99-bc48-5ea894290b36" containerName="cleanup" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743613 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" containerName="payload-processing" Oct 30 19:30:45.743607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743617 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" containerName="payload-processing" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743625 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743629 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743636 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743640 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de2cfb0-4f3c-4684-b68f-8e87b985bd7a" containerName="cleanup" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743724 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e042745e-ef0d-4f87-9b6c-fffa8c3d81b3" containerName="payload-processing" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743730 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ce8fe9-fce6-4dba-93c7-4efd9712e9cf" containerName="istio-proxy" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743738 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b9e5cd-a841-42c8-a898-3c0206a42ed1" containerName="cleanup" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743745 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c81b220-9f81-41ba-b0ea-53ee0b873997" containerName="revoke-keys" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743751 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee35881-5f88-4564-bc41-a50e1fbf40da" containerName="cleanup" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743757 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fbeb12-4da0-473b-b064-071f95348b7b" containerName="cleanup" Oct 30 19:30:45.744294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.743764 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5531d96f-29f5-488c-bf3a-b3f1ef5a719a" containerName="cleanup" Oct 30 19:30:45.748705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.748683 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.798827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.798799 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-69d429-openshift-default-dockercfg-frl4m\"" Oct 30 19:30:45.848942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.848905 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:30:45.872016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.871982 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872104 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872139 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872227 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872279 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872351 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jjn\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872431 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872497 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.872552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.872544 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.973712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973674 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.973901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973725 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.973901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973756 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.973901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973798 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.973901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973895 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jjn\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973939 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.973997 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974047 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974075 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974253 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974284 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974398 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974549 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.974599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.974587 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.976355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.976328 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:45.976484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:45.976467 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:46.032922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.032838 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:46.033079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.033059 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jjn\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\") pod \"e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:46.075275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.075249 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:46.312539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.312510 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:30:46.378710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378671 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.378881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378720 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.378881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378756 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.378881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378794 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.378881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378831 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.378881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378872 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.379175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378934 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.379175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.378993 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.379175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379021 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\") pod \"83da78a7-ce06-4c04-8801-aa26491c3e26\" (UID: \"83da78a7-ce06-4c04-8801-aa26491c3e26\") " Oct 30 19:30:46.379175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379097 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:46.379430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379351 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:46.379782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379508 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.379782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379534 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.379782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379542 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:46.379782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.379747 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:46.380409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.380387 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:46.381798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.381766 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:46.381924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.381825 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "kube-api-access-ncgnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:46.381924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.381874 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:30:46.382037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.381988 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token" pod "83da78a7-ce06-4c04-8801-aa26491c3e26" (UID: "83da78a7-ce06-4c04-8801-aa26491c3e26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:46.480671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480637 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480665 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480676 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480685 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480694 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83da78a7-ce06-4c04-8801-aa26491c3e26-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480703 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgnp\" (UniqueName: \"kubernetes.io/projected/83da78a7-ce06-4c04-8801-aa26491c3e26-kube-api-access-ncgnp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.480897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.480712 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83da78a7-ce06-4c04-8801-aa26491c3e26-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:46.512551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.512524 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:30:46.514155 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:46.514131 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5a4780_67c7_4c14_b14a_c9a6acdf9e11.slice/crio-c9b4140b7e3b389e0662aed52054e639f87f5b03dc590ff5c33447d0c41cb1b0 WatchSource:0}: Error finding container c9b4140b7e3b389e0662aed52054e639f87f5b03dc590ff5c33447d0c41cb1b0: Status 404 returned error can't find the container with id c9b4140b7e3b389e0662aed52054e639f87f5b03dc590ff5c33447d0c41cb1b0 Oct 30 19:30:46.516626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.516594 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:46.516700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.516660 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:46.516700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:46.516691 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:30:47.151558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.151517 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerStarted","Data":"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc"} Oct 30 19:30:47.152002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.151586 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerStarted","Data":"c9b4140b7e3b389e0662aed52054e639f87f5b03dc590ff5c33447d0c41cb1b0"} Oct 30 19:30:47.153340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.153311 4789 generic.go:356] "Generic (PLEG): container finished" podID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerID="88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f" exitCode=0 Oct 30 19:30:47.153461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.153426 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" Oct 30 19:30:47.153521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.153455 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" event={"ID":"83da78a7-ce06-4c04-8801-aa26491c3e26","Type":"ContainerDied","Data":"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f"} Oct 30 19:30:47.153521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.153515 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6" event={"ID":"83da78a7-ce06-4c04-8801-aa26491c3e26","Type":"ContainerDied","Data":"c6145d98d3b748c9b326f7c9fe864b55a098f6fbcfe993fa71ac040466eb60e2"} Oct 30 19:30:47.153620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.153550 4789 scope.go:122] "RemoveContainer" containerID="88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f" Oct 30 19:30:47.164853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.164834 4789 scope.go:122] "RemoveContainer" containerID="88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f" Oct 30 19:30:47.165170 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:47.165144 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f\": container with ID starting with 88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f not found: ID does not exist" containerID="88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f" Oct 30 19:30:47.165296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.165177 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f"} err="failed to get container status \"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f\": rpc error: code = NotFound desc = could not find container \"88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f\": container with ID starting with 88ebdc7cf95748fc10b9303aedbf4a1aef476ce9886fd166b29c736548a8dc3f not found: ID does not exist" Oct 30 19:30:47.274719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.274616 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podStartSLOduration=2.274602966 podStartE2EDuration="2.274602966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:47.270385713 +0000 UTC m=+2683.772296199" watchObservedRunningTime="2026-10-30 19:30:47.274602966 +0000 UTC m=+2683.776513453" Oct 30 19:30:47.409095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.409065 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:30:47.431368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.431342 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-6g9z6"] Oct 30 19:30:47.667943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:47.667863 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" path="/var/lib/kubelet/pods/83da78a7-ce06-4c04-8801-aa26491c3e26/volumes" Oct 30 19:30:48.075484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:48.075452 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:48.081302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:48.081279 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:48.160914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:48.160883 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:48.161803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:48.161784 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:30:48.197705 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:30:48.197674 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:30:54.279203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.279169 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:30:54.279704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.279643 4789 cpu_manager.go:414] "containerMap: removing container" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" Oct 30 19:30:54.279704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.279657 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" Oct 30 19:30:54.279781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.279765 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="83da78a7-ce06-4c04-8801-aa26491c3e26" containerName="istio-proxy" Oct 30 19:30:54.284706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.284685 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.364529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.364486 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.364745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.364703 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wx5\" (UniqueName: \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.381812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.381778 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-69d429\"" Oct 30 19:30:54.381812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.381778 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-69d429-dockercfg-d9xsb\"" Oct 30 19:30:54.386014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.385985 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:30:54.440644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.440602 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:30:54.444619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.444597 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.465936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.465914 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wx5\" (UniqueName: \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.466055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.466005 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.466725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.466699 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.500277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.500237 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:30:54.525279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.525247 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wx5\" (UniqueName: \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\") pod \"payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.567262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.567184 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875rp\" (UniqueName: \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.567379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.567314 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.599740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.599719 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:54.668306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.668115 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875rp\" (UniqueName: \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.668306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.668247 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.669889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.669092 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.726568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.726535 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875rp\" (UniqueName: \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\") pod \"payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.759043 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.759014 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:54.976089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:54.976062 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:30:54.978406 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:54.978380 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cca9a3_f612_4f2a_87c7_688892c618d7.slice/crio-fad71ccac3f681a5ecf3204984b4d9bfb91b675c5f3515c56862a406cdf79de5 WatchSource:0}: Error finding container fad71ccac3f681a5ecf3204984b4d9bfb91b675c5f3515c56862a406cdf79de5: Status 404 returned error can't find the container with id fad71ccac3f681a5ecf3204984b4d9bfb91b675c5f3515c56862a406cdf79de5 Oct 30 19:30:55.144285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.144258 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:30:55.148437 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:30:55.148407 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6231fa9_65ba_4d93_8aa9_4161e52d1ffc.slice/crio-a0890122eac48d565a75b8899adcca3cdf4d306166dac3ac0d88bdb9adcb6aea WatchSource:0}: Error finding container a0890122eac48d565a75b8899adcca3cdf4d306166dac3ac0d88bdb9adcb6aea: Status 404 returned error can't find the container with id a0890122eac48d565a75b8899adcca3cdf4d306166dac3ac0d88bdb9adcb6aea Oct 30 19:30:55.205422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.205391 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" event={"ID":"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc","Type":"ContainerStarted","Data":"a0890122eac48d565a75b8899adcca3cdf4d306166dac3ac0d88bdb9adcb6aea"} Oct 30 19:30:55.207730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.207706 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" event={"ID":"a9cca9a3-f612-4f2a-87c7-688892c618d7","Type":"ContainerStarted","Data":"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034"} Oct 30 19:30:55.207950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.207932 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" event={"ID":"a9cca9a3-f612-4f2a-87c7-688892c618d7","Type":"ContainerStarted","Data":"fad71ccac3f681a5ecf3204984b4d9bfb91b675c5f3515c56862a406cdf79de5"} Oct 30 19:30:55.208126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.208108 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:30:55.318331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:55.318258 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" podStartSLOduration=1.318241847 podStartE2EDuration="1.318241847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:55.317927161 +0000 UTC m=+2691.819837648" watchObservedRunningTime="2026-10-30 19:30:55.318241847 +0000 UTC m=+2691.820152328" Oct 30 19:30:56.223614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:56.223559 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" event={"ID":"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc","Type":"ContainerStarted","Data":"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981"} Oct 30 19:30:56.224747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:56.224719 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:30:56.335879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:56.335821 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" podStartSLOduration=2.335805704 podStartE2EDuration="2.335805704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.334695149 +0000 UTC m=+2692.836605635" watchObservedRunningTime="2026-10-30 19:30:56.335805704 +0000 UTC m=+2692.837716189" Oct 30 19:30:57.380159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:57.380128 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56"] Oct 30 19:30:57.427748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:57.427710 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4833f3-6rx56"] Oct 30 19:30:57.667486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:30:57.667408 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c81b220-9f81-41ba-b0ea-53ee0b873997" path="/var/lib/kubelet/pods/5c81b220-9f81-41ba-b0ea-53ee0b873997/volumes" Oct 30 19:31:01.931337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:01.931302 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:31:01.935805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:01.935784 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:01.989419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:01.989387 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e73f1a-openshift-default-dockercfg-l74qp\"" Oct 30 19:31:02.039981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.039932 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:31:02.046870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.046832 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.046880 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.046909 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.046972 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.047049 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.047096 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.047144 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.047176 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qv9t\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.047347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.047238 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148126 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148192 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148242 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148271 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qv9t\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148304 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148363 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148393 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148416 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148490 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148621 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148873 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.148948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.148924 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.149063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.149019 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.149183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.149161 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.150840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.150813 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.151116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.151096 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.207919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.207878 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.208485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.208463 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qv9t\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\") pod \"e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.249431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.249399 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:02.684422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.684396 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:31:02.686687 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:02.686662 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ddb744_c12f_463e_912c_92a1b2e0ca4a.slice/crio-54d9d749e160458ea6bfcc938dd1e33cb3a19634bb6fbca4c925dbc5487cc5e6 WatchSource:0}: Error finding container 54d9d749e160458ea6bfcc938dd1e33cb3a19634bb6fbca4c925dbc5487cc5e6: Status 404 returned error can't find the container with id 54d9d749e160458ea6bfcc938dd1e33cb3a19634bb6fbca4c925dbc5487cc5e6 Oct 30 19:31:02.689013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.688980 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:02.689106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.689044 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:02.689106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:02.689073 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:31:03.272471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:03.272442 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerStarted","Data":"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef"} Oct 30 19:31:03.272872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:03.272493 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerStarted","Data":"54d9d749e160458ea6bfcc938dd1e33cb3a19634bb6fbca4c925dbc5487cc5e6"} Oct 30 19:31:03.387814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:03.387766 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podStartSLOduration=2.387752746 podStartE2EDuration="2.387752746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:03.38640192 +0000 UTC m=+2699.888312418" watchObservedRunningTime="2026-10-30 19:31:03.387752746 +0000 UTC m=+2699.889663232" Oct 30 19:31:04.250201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.250169 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:04.255865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.255842 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:04.279070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.279051 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:04.279846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.279827 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:31:04.352976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.352954 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:31:04.368452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:04.368427 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:31:06.129009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:06.128973 4789 scope.go:122] "RemoveContainer" containerID="65dc37337101f73a4549a7c7eda72b5b2495c1f0f70118251b634af3eb56b4f4" Oct 30 19:31:06.226922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:06.226896 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:31:08.240171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:08.240137 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:31:18.145978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.145936 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:31:18.151699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.151675 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.196018 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:31:18.195984 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:31:18.206282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.206256 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:31:18.211129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.211104 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dkw\" (UniqueName: \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.211275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.211147 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.249510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.249451 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e73f1a\"" Oct 30 19:31:18.249680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.249581 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e73f1a-dockercfg-s8497\"" Oct 30 19:31:18.312340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.312306 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dkw\" (UniqueName: \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.312614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.312583 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.313577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.313551 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.334145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.334119 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:31:18.338500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.338480 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.370475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.370444 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dkw\" (UniqueName: \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\") pod \"payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.393547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.393523 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:31:18.413493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.413424 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdpt\" (UniqueName: \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.413493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.413479 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.463931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.463904 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:18.518660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.517612 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdpt\" (UniqueName: \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.520695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.520667 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.520891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.520861 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.576611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.576583 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdpt\" (UniqueName: \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\") pod \"payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.652017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.651989 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:18.843006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:18.842977 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:31:18.845690 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:18.845663 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93142e47_3f86_4a30_b03a_c5e205e37f46.slice/crio-e7734ed96c4738adbd6cd3b3fe1415c538394c82e2531d32f50263096f9df385 WatchSource:0}: Error finding container e7734ed96c4738adbd6cd3b3fe1415c538394c82e2531d32f50263096f9df385: Status 404 returned error can't find the container with id e7734ed96c4738adbd6cd3b3fe1415c538394c82e2531d32f50263096f9df385 Oct 30 19:31:19.032740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.032702 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:31:19.035767 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:19.035738 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14ab7b6_3a4d_4ac8_84f4_078a62a540ee.slice/crio-1ebf263e10899ee1f0b49a7c419a290328e71f0e769b8955bbfc56687a4aaad0 WatchSource:0}: Error finding container 1ebf263e10899ee1f0b49a7c419a290328e71f0e769b8955bbfc56687a4aaad0: Status 404 returned error can't find the container with id 1ebf263e10899ee1f0b49a7c419a290328e71f0e769b8955bbfc56687a4aaad0 Oct 30 19:31:19.379716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.379628 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" event={"ID":"93142e47-3f86-4a30-b03a-c5e205e37f46","Type":"ContainerStarted","Data":"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd"} Oct 30 19:31:19.379716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.379693 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:19.380191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.379720 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" event={"ID":"93142e47-3f86-4a30-b03a-c5e205e37f46","Type":"ContainerStarted","Data":"e7734ed96c4738adbd6cd3b3fe1415c538394c82e2531d32f50263096f9df385"} Oct 30 19:31:19.381650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.381623 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" event={"ID":"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee","Type":"ContainerStarted","Data":"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0"} Oct 30 19:31:19.381763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.381673 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" event={"ID":"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee","Type":"ContainerStarted","Data":"1ebf263e10899ee1f0b49a7c419a290328e71f0e769b8955bbfc56687a4aaad0"} Oct 30 19:31:19.381763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.381702 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:19.490673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.490628 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" podStartSLOduration=1.490614694 podStartE2EDuration="1.490614694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.489529882 +0000 UTC m=+2715.991440371" watchObservedRunningTime="2026-10-30 19:31:19.490614694 +0000 UTC m=+2715.992525179" Oct 30 19:31:19.601369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:19.601298 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" podStartSLOduration=1.6012805540000001 podStartE2EDuration="1.601280554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.599323243 +0000 UTC m=+2716.101233729" watchObservedRunningTime="2026-10-30 19:31:19.601280554 +0000 UTC m=+2716.103191054" Oct 30 19:31:20.978700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:20.978670 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682"] Oct 30 19:31:20.982652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:20.982633 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.039396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.039359 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682"] Oct 30 19:31:21.062361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.062334 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz877\" (UniqueName: \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.062495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.062373 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.062495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.062464 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.079818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.079799 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:21.079943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.079859 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.079943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.079889 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:31:21.079943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.079904 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.163486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.163458 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz877\" (UniqueName: \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.163606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.163498 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.163606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.163550 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.164254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.164231 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.165842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.165822 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.222587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.222558 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz877\" (UniqueName: \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\") pod \"maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.297453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.297400 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:21.675049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:21.674968 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682"] Oct 30 19:31:21.677464 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:21.677438 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc063ae2f_e505_46eb_9027_f34fe997a826.slice/crio-60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7 WatchSource:0}: Error finding container 60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7: Status 404 returned error can't find the container with id 60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7 Oct 30 19:31:22.402143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:22.402059 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" event={"ID":"c063ae2f-e505-46eb-9027-f34fe997a826","Type":"ContainerStarted","Data":"72955da42693e7df070569c5d9a335b8b900d43319779efae00a1d5b2bc0234b"} Oct 30 19:31:22.402143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:22.402128 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" event={"ID":"c063ae2f-e505-46eb-9027-f34fe997a826","Type":"ContainerStarted","Data":"60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7"} Oct 30 19:31:22.513959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:22.513900 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" podStartSLOduration=2.513883165 podStartE2EDuration="2.513883165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:22.512065719 +0000 UTC m=+2719.013976206" watchObservedRunningTime="2026-10-30 19:31:22.513883165 +0000 UTC m=+2719.015793648" Oct 30 19:31:23.409599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:23.409554 4789 generic.go:356] "Generic (PLEG): container finished" podID="c063ae2f-e505-46eb-9027-f34fe997a826" containerID="72955da42693e7df070569c5d9a335b8b900d43319779efae00a1d5b2bc0234b" exitCode=0 Oct 30 19:31:23.410003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:23.409633 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" event={"ID":"c063ae2f-e505-46eb-9027-f34fe997a826","Type":"ContainerDied","Data":"72955da42693e7df070569c5d9a335b8b900d43319779efae00a1d5b2bc0234b"} Oct 30 19:31:24.566586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.566559 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:24.698792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.698766 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\" (UniqueName: \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\") pod \"c063ae2f-e505-46eb-9027-f34fe997a826\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " Oct 30 19:31:24.698982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.698802 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\") pod \"c063ae2f-e505-46eb-9027-f34fe997a826\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " Oct 30 19:31:24.698982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.698863 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\") pod \"c063ae2f-e505-46eb-9027-f34fe997a826\" (UID: \"c063ae2f-e505-46eb-9027-f34fe997a826\") " Oct 30 19:31:24.699508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.699482 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca" pod "c063ae2f-e505-46eb-9027-f34fe997a826" (UID: "c063ae2f-e505-46eb-9027-f34fe997a826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:24.705343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.705311 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca" pod "c063ae2f-e505-46eb-9027-f34fe997a826" (UID: "c063ae2f-e505-46eb-9027-f34fe997a826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:24.707346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.707320 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877" pod "c063ae2f-e505-46eb-9027-f34fe997a826" (UID: "c063ae2f-e505-46eb-9027-f34fe997a826"). InnerVolumeSpecName "kube-api-access-rz877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:24.801259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.801232 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz877\" (UniqueName: \"kubernetes.io/projected/c063ae2f-e505-46eb-9027-f34fe997a826-kube-api-access-rz877\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.801259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.801256 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:24.801259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:24.801267 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c063ae2f-e505-46eb-9027-f34fe997a826-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:25.019406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:25.019325 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:31:25.118921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:25.118888 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:31:25.426862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:25.426776 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" Oct 30 19:31:25.427035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:25.426780 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682" event={"ID":"c063ae2f-e505-46eb-9027-f34fe997a826","Type":"ContainerDied","Data":"60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7"} Oct 30 19:31:25.427035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:25.426903 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60dd9ac32e93d639ff3551ad8660a0c13223923f59c90350952e63d252c880c7" Oct 30 19:31:30.019747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.019709 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" podUID="b8438e49-6f00-4dc7-8253-9ff738f80437" containerName="payload-processing" containerID="cri-o://b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375" gracePeriod=25 Oct 30 19:31:30.120148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.120116 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" podUID="1b7c23c1-b903-4930-ae24-02b863c157cf" containerName="payload-pre-processing" containerID="cri-o://a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50" gracePeriod=25 Oct 30 19:31:30.285527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.285124 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:31:30.376340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.376318 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:31:30.391232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.391191 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:31:30.391333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.391301 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:31:30.460848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.460823 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\") pod \"b8438e49-6f00-4dc7-8253-9ff738f80437\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " Oct 30 19:31:30.461028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.460877 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\" (UniqueName: \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\") pod \"b8438e49-6f00-4dc7-8253-9ff738f80437\" (UID: \"b8438e49-6f00-4dc7-8253-9ff738f80437\") " Oct 30 19:31:30.461303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461276 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume" pod "b8438e49-6f00-4dc7-8253-9ff738f80437" (UID: "b8438e49-6f00-4dc7-8253-9ff738f80437"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:30.461686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461660 4789 generic.go:356] "Generic (PLEG): container finished" podID="b8438e49-6f00-4dc7-8253-9ff738f80437" containerID="b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375" exitCode=0 Oct 30 19:31:30.461768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461713 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" Oct 30 19:31:30.461768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461733 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" event={"ID":"b8438e49-6f00-4dc7-8253-9ff738f80437","Type":"ContainerDied","Data":"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375"} Oct 30 19:31:30.461865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461785 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc" event={"ID":"b8438e49-6f00-4dc7-8253-9ff738f80437","Type":"ContainerDied","Data":"5ae068d8f636d55a3a1ad09e89d32d2c95e5c0ca7b71651b0c8ddb0167638821"} Oct 30 19:31:30.461865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.461821 4789 scope.go:122] "RemoveContainer" containerID="b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375" Oct 30 19:31:30.463045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.463023 4789 generic.go:356] "Generic (PLEG): container finished" podID="1b7c23c1-b903-4930-ae24-02b863c157cf" containerID="a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50" exitCode=0 Oct 30 19:31:30.463144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.463061 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" Oct 30 19:31:30.463144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.463110 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" event={"ID":"1b7c23c1-b903-4930-ae24-02b863c157cf","Type":"ContainerDied","Data":"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50"} Oct 30 19:31:30.463262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.463153 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb" event={"ID":"1b7c23c1-b903-4930-ae24-02b863c157cf","Type":"ContainerDied","Data":"4b3c5743898279fd22ca2b42125f5f558e6c730d2b9cfe77b30026b37bf315c7"} Oct 30 19:31:30.463307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.463270 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d" pod "b8438e49-6f00-4dc7-8253-9ff738f80437" (UID: "b8438e49-6f00-4dc7-8253-9ff738f80437"). InnerVolumeSpecName "kube-api-access-dwf8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:30.471375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.471358 4789 scope.go:122] "RemoveContainer" containerID="b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375" Oct 30 19:31:30.471637 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:31:30.471617 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375\": container with ID starting with b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375 not found: ID does not exist" containerID="b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375" Oct 30 19:31:30.471694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.471644 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375"} err="failed to get container status \"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375\": rpc error: code = NotFound desc = could not find container \"b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375\": container with ID starting with b60472abbb2300c09f81890ece3072e01c2dfb3b8a5090e2e853d64e8ea4c375 not found: ID does not exist" Oct 30 19:31:30.471694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.471669 4789 scope.go:122] "RemoveContainer" containerID="a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50" Oct 30 19:31:30.479985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.479970 4789 scope.go:122] "RemoveContainer" containerID="a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50" Oct 30 19:31:30.480261 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:31:30.480244 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50\": container with ID starting with a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50 not found: ID does not exist" containerID="a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50" Oct 30 19:31:30.480324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.480267 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50"} err="failed to get container status \"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50\": rpc error: code = NotFound desc = could not find container \"a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50\": container with ID starting with a5f60bbb97861a73dd5aefbca1a5622398c1ac27cc005d348f74535eabb87d50 not found: ID does not exist" Oct 30 19:31:30.562384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.562317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\" (UniqueName: \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\") pod \"1b7c23c1-b903-4930-ae24-02b863c157cf\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " Oct 30 19:31:30.562527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.562414 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\") pod \"1b7c23c1-b903-4930-ae24-02b863c157cf\" (UID: \"1b7c23c1-b903-4930-ae24-02b863c157cf\") " Oct 30 19:31:30.562829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.562802 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume" pod "1b7c23c1-b903-4930-ae24-02b863c157cf" (UID: "1b7c23c1-b903-4930-ae24-02b863c157cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:30.562981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.562840 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8438e49-6f00-4dc7-8253-9ff738f80437-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:30.563078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.563057 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwf8d\" (UniqueName: \"kubernetes.io/projected/b8438e49-6f00-4dc7-8253-9ff738f80437-kube-api-access-dwf8d\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:30.564943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.564922 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr" pod "1b7c23c1-b903-4930-ae24-02b863c157cf" (UID: "1b7c23c1-b903-4930-ae24-02b863c157cf"). InnerVolumeSpecName "kube-api-access-tvrdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:30.663908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.663869 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7c23c1-b903-4930-ae24-02b863c157cf-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:30.663908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.663897 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrdr\" (UniqueName: \"kubernetes.io/projected/1b7c23c1-b903-4930-ae24-02b863c157cf-kube-api-access-tvrdr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:30.933124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.933052 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:31:30.936959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:30.936934 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f3b5af-65876cdb4c-rtlmb"] Oct 30 19:31:31.038701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:31.038677 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:31:31.045070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:31.045049 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-74f84b4c94-v9rvc"] Oct 30 19:31:31.670142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:31.670106 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7c23c1-b903-4930-ae24-02b863c157cf" path="/var/lib/kubelet/pods/1b7c23c1-b903-4930-ae24-02b863c157cf/volumes" Oct 30 19:31:31.671103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:31.671081 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8438e49-6f00-4dc7-8253-9ff738f80437" path="/var/lib/kubelet/pods/b8438e49-6f00-4dc7-8253-9ff738f80437/volumes" Oct 30 19:31:34.487384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:34.487351 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5_4eb356a8-a99a-4b18-b5c1-47212269ea45/payload-processing/0.log" Oct 30 19:31:36.747094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.747059 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn"] Oct 30 19:31:36.748198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748166 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c063ae2f-e505-46eb-9027-f34fe997a826" containerName="revoke-keys" Oct 30 19:31:36.748198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748192 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c063ae2f-e505-46eb-9027-f34fe997a826" containerName="revoke-keys" Oct 30 19:31:36.748392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748301 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1b7c23c1-b903-4930-ae24-02b863c157cf" containerName="payload-pre-processing" Oct 30 19:31:36.748392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748310 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7c23c1-b903-4930-ae24-02b863c157cf" containerName="payload-pre-processing" Oct 30 19:31:36.748392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748331 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b8438e49-6f00-4dc7-8253-9ff738f80437" containerName="payload-processing" Oct 30 19:31:36.748392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748339 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8438e49-6f00-4dc7-8253-9ff738f80437" containerName="payload-processing" Oct 30 19:31:36.748611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748595 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c063ae2f-e505-46eb-9027-f34fe997a826" containerName="revoke-keys" Oct 30 19:31:36.748674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748633 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7c23c1-b903-4930-ae24-02b863c157cf" containerName="payload-pre-processing" Oct 30 19:31:36.748674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.748651 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8438e49-6f00-4dc7-8253-9ff738f80437" containerName="payload-processing" Oct 30 19:31:36.754844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.754819 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.805885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.805858 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn"] Oct 30 19:31:36.830332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.830299 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.830466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.830443 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.830537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.830484 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9jl\" (UniqueName: \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.848673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.848645 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:36.848805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.848680 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:36.848805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.848784 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:31:36.848927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.848812 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:36.931403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.931369 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.931564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.931410 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9jl\" (UniqueName: \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.931564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.931526 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.932335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.932311 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.933726 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.933696 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:36.989709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:36.989686 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9jl\" (UniqueName: \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\") pod \"maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:37.068331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:37.068273 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:37.443018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:37.442936 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn"] Oct 30 19:31:37.446260 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:37.446231 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c777293_6889_4b02_b197_82be02f56a02.slice/crio-a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4 WatchSource:0}: Error finding container a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4: Status 404 returned error can't find the container with id a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4 Oct 30 19:31:37.515675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:37.515647 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" event={"ID":"6c777293-6889-4b02-b197-82be02f56a02","Type":"ContainerStarted","Data":"a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4"} Oct 30 19:31:38.523066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:38.522970 4789 generic.go:356] "Generic (PLEG): container finished" podID="6c777293-6889-4b02-b197-82be02f56a02" containerID="2b2a0a5f23bfbd409c059f65767fe4d82eef5df5daf1b6eee1de8f50c392db79" exitCode=0 Oct 30 19:31:38.523461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:38.523061 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" event={"ID":"6c777293-6889-4b02-b197-82be02f56a02","Type":"ContainerDied","Data":"2b2a0a5f23bfbd409c059f65767fe4d82eef5df5daf1b6eee1de8f50c392db79"} Oct 30 19:31:39.675477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.675452 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:39.761009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.760983 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\" (UniqueName: \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\") pod \"6c777293-6889-4b02-b197-82be02f56a02\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " Oct 30 19:31:39.761179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.761017 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\") pod \"6c777293-6889-4b02-b197-82be02f56a02\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " Oct 30 19:31:39.761179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.761103 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\") pod \"6c777293-6889-4b02-b197-82be02f56a02\" (UID: \"6c777293-6889-4b02-b197-82be02f56a02\") " Oct 30 19:31:39.761529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.761504 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca" pod "6c777293-6889-4b02-b197-82be02f56a02" (UID: "6c777293-6889-4b02-b197-82be02f56a02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:39.763058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.763037 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca" pod "6c777293-6889-4b02-b197-82be02f56a02" (UID: "6c777293-6889-4b02-b197-82be02f56a02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:39.763114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.763069 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl" pod "6c777293-6889-4b02-b197-82be02f56a02" (UID: "6c777293-6889-4b02-b197-82be02f56a02"). InnerVolumeSpecName "kube-api-access-ft9jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:39.862049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.861980 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9jl\" (UniqueName: \"kubernetes.io/projected/6c777293-6889-4b02-b197-82be02f56a02-kube-api-access-ft9jl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:39.862049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.862007 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c777293-6889-4b02-b197-82be02f56a02-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:39.862049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:39.862018 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c777293-6889-4b02-b197-82be02f56a02-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:40.538137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:40.538110 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" Oct 30 19:31:40.538323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:40.538148 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn" event={"ID":"6c777293-6889-4b02-b197-82be02f56a02","Type":"ContainerDied","Data":"a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4"} Oct 30 19:31:40.538323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:40.538194 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a017151ff9cf67e464aeb8181a53e05d75013b6d15a52bacc1f4973583a68df4" Oct 30 19:31:47.669144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.669064 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb"] Oct 30 19:31:47.671670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.669576 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6c777293-6889-4b02-b197-82be02f56a02" containerName="revoke-keys" Oct 30 19:31:47.671670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.669590 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c777293-6889-4b02-b197-82be02f56a02" containerName="revoke-keys" Oct 30 19:31:47.671670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.669703 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c777293-6889-4b02-b197-82be02f56a02" containerName="revoke-keys" Oct 30 19:31:47.672815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.672801 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.726905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.726872 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb"] Oct 30 19:31:47.728617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.728597 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.728787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.728763 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.728849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.728812 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.770340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.770316 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:47.770470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.770411 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:31:47.770470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.770449 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:31:47.770470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.770469 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:31:47.829676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.829650 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.829788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.829683 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.829788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.829760 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.830438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.830417 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.831934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.831913 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.887634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.887610 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\") pod \"maas-api-revoke-sub-602bbbac-qrqhb\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:47.984895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:47.984870 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:48.197044 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:31:48.197009 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:31:48.361015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:48.360989 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb"] Oct 30 19:31:48.363158 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:31:48.363130 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72708a26_781a_4c4a_af03_7b623bef1678.slice/crio-0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d WatchSource:0}: Error finding container 0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d: Status 404 returned error can't find the container with id 0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d Oct 30 19:31:48.587268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:48.587233 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" event={"ID":"72708a26-781a-4c4a-af03-7b623bef1678","Type":"ContainerStarted","Data":"0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d"} Oct 30 19:31:49.594807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:49.594709 4789 generic.go:356] "Generic (PLEG): container finished" podID="72708a26-781a-4c4a-af03-7b623bef1678" containerID="81b3940187a0e4f5d3f242b3646a09b6be4f592b2c58d85cccd88743cf2ff2f1" exitCode=0 Oct 30 19:31:49.594807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:49.594759 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" event={"ID":"72708a26-781a-4c4a-af03-7b623bef1678","Type":"ContainerDied","Data":"81b3940187a0e4f5d3f242b3646a09b6be4f592b2c58d85cccd88743cf2ff2f1"} Oct 30 19:31:50.750933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.750908 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:50.862240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.862154 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\") pod \"72708a26-781a-4c4a-af03-7b623bef1678\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " Oct 30 19:31:50.862938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.862272 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\") pod \"72708a26-781a-4c4a-af03-7b623bef1678\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " Oct 30 19:31:50.862938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.862308 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\") pod \"72708a26-781a-4c4a-af03-7b623bef1678\" (UID: \"72708a26-781a-4c4a-af03-7b623bef1678\") " Oct 30 19:31:50.862938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.862729 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca" pod "72708a26-781a-4c4a-af03-7b623bef1678" (UID: "72708a26-781a-4c4a-af03-7b623bef1678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:50.865294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.865264 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca" pod "72708a26-781a-4c4a-af03-7b623bef1678" (UID: "72708a26-781a-4c4a-af03-7b623bef1678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:50.865587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.865541 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2" pod "72708a26-781a-4c4a-af03-7b623bef1678" (UID: "72708a26-781a-4c4a-af03-7b623bef1678"). InnerVolumeSpecName "kube-api-access-l5cv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:50.963723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.963695 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5cv2\" (UniqueName: \"kubernetes.io/projected/72708a26-781a-4c4a-af03-7b623bef1678-kube-api-access-l5cv2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:50.963723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.963720 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:50.963918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:50.963733 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72708a26-781a-4c4a-af03-7b623bef1678-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:51.609807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:51.609777 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" Oct 30 19:31:51.610000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:51.609816 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb" event={"ID":"72708a26-781a-4c4a-af03-7b623bef1678","Type":"ContainerDied","Data":"0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d"} Oct 30 19:31:51.610000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:31:51.609908 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bb2b0bcdd3e5bee1f33d4c774aed4db382e20f5e240526dfc9aa9fcc522a59d" Oct 30 19:32:06.193958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:06.193919 4789 scope.go:122] "RemoveContainer" containerID="edfd022582243466b73cbaeb160dc3353e64b044cc3b1ed37c7ce95213495de8" Oct 30 19:32:06.212342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:06.212309 4789 scope.go:122] "RemoveContainer" containerID="0eb11c147c646cb0169d7769b098f5332673604f850e7ce55ca9d80e1640b7cd" Oct 30 19:32:08.320339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:08.320306 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682"] Oct 30 19:32:08.366026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:08.365994 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c16ba7-kd682"] Oct 30 19:32:09.668985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:09.668951 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c063ae2f-e505-46eb-9027-f34fe997a826" path="/var/lib/kubelet/pods/c063ae2f-e505-46eb-9027-f34fe997a826/volumes" Oct 30 19:32:17.781360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.781324 4789 generic.go:356] "Generic (PLEG): container finished" podID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerID="87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc" exitCode=0 Oct 30 19:32:17.781788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.781403 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerDied","Data":"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc"} Oct 30 19:32:17.781836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.781792 4789 scope.go:122] "RemoveContainer" containerID="87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc" Oct 30 19:32:17.783134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.783110 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:17.783199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.783175 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:17.783265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:17.783240 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:18.161366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:18.161290 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:32:18.196720 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:32:18.196681 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:32:18.789393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:18.789359 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerStarted","Data":"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c"} Oct 30 19:32:19.075492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:19.075405 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:32:19.076687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:19.076662 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:32:19.076806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:19.076714 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:32:20.076417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:20.076376 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:32:20.076830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:20.076443 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:32:21.076388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:21.076353 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:32:21.076571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:21.076407 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:32:22.075681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.075640 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:32:22.075874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.075716 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:32:22.526879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.526843 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:32:22.527498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.527480 4789 cpu_manager.go:414] "containerMap: removing container" podUID="72708a26-781a-4c4a-af03-7b623bef1678" containerName="revoke-keys" Oct 30 19:32:22.527560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.527499 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72708a26-781a-4c4a-af03-7b623bef1678" containerName="revoke-keys" Oct 30 19:32:22.527619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.527610 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="72708a26-781a-4c4a-af03-7b623bef1678" containerName="revoke-keys" Oct 30 19:32:22.531117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.531099 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.555933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.555903 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cd72\" (UniqueName: \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.556072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.556058 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.582269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.582240 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f3b5af\"" Oct 30 19:32:22.637695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.637660 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f3b5af-dockercfg-4dvwv\"" Oct 30 19:32:22.638986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.638944 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:32:22.657526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.657494 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cd72\" (UniqueName: \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.657691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.657602 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.658283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.658262 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.725003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.724970 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cd72\" (UniqueName: \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\") pod \"payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:22.844269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:22.844152 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:23.075986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.075951 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:32:23.076170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.076005 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:32:23.252917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.252881 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:32:23.256565 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:32:23.256537 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353d9a4a_0ee3_46d1_9f8a_7ae79eb10fc2.slice/crio-86d0370d258a31ab60adc617c52444749eec4e90ac339579928ad594cb79c526 WatchSource:0}: Error finding container 86d0370d258a31ab60adc617c52444749eec4e90ac339579928ad594cb79c526: Status 404 returned error can't find the container with id 86d0370d258a31ab60adc617c52444749eec4e90ac339579928ad594cb79c526 Oct 30 19:32:23.622962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.622867 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc"] Oct 30 19:32:23.628566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.628530 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.670059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.669984 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.670059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.670030 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbcn\" (UniqueName: \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.670317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.670166 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.675104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.675077 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc"] Oct 30 19:32:23.678508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.678461 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:23.678508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.678463 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:32:23.721738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.721440 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:32:23.721738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.721452 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:32:23.771846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.771805 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.772049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.771871 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbcn\" (UniqueName: \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.772049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.771962 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.772827 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.772792 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.774403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.774375 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.831734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.831695 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbcn\" (UniqueName: \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\") pod \"maas-api-revoke-sub-4dcc70a0-k4qfc\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.856680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.856642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" event={"ID":"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2","Type":"ContainerStarted","Data":"e43f270beee1b6f16f7e2ec8b40f45b1499388a866738551586dff89e37eadb7"} Oct 30 19:32:23.856846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.856706 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" event={"ID":"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2","Type":"ContainerStarted","Data":"86d0370d258a31ab60adc617c52444749eec4e90ac339579928ad594cb79c526"} Oct 30 19:32:23.856846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.856740 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:23.951622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.951587 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:23.973499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:23.973422 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" podStartSLOduration=1.9734011360000001 podStartE2EDuration="1.973401136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:23.969321249 +0000 UTC m=+2780.471231734" watchObservedRunningTime="2026-10-30 19:32:23.973401136 +0000 UTC m=+2780.475311624" Oct 30 19:32:24.082144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.082116 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:32:24.082871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.082633 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:32:24.083419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.083398 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:32:24.337236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.333686 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc"] Oct 30 19:32:24.865373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.865336 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" event={"ID":"92672c4e-204a-4e2e-a9ad-51f08dff8f80","Type":"ContainerStarted","Data":"5ed15723f9c4ab8d52660ca362a7a12e14c2c3f890458f9cb51544580ced922a"} Oct 30 19:32:24.865802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.865395 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" event={"ID":"92672c4e-204a-4e2e-a9ad-51f08dff8f80","Type":"ContainerStarted","Data":"a31d7d99d39d8cb19fbbc80f7ab0ab158f6f99fb7e42d1158053831d45242407"} Oct 30 19:32:24.976065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:24.976011 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" podStartSLOduration=1.975996983 podStartE2EDuration="1.975996983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:24.975198488 +0000 UTC m=+2781.477108976" watchObservedRunningTime="2026-10-30 19:32:24.975996983 +0000 UTC m=+2781.477907469" Oct 30 19:32:25.872729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:25.872689 4789 generic.go:356] "Generic (PLEG): container finished" podID="92672c4e-204a-4e2e-a9ad-51f08dff8f80" containerID="5ed15723f9c4ab8d52660ca362a7a12e14c2c3f890458f9cb51544580ced922a" exitCode=0 Oct 30 19:32:25.873270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:25.872730 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" event={"ID":"92672c4e-204a-4e2e-a9ad-51f08dff8f80","Type":"ContainerDied","Data":"5ed15723f9c4ab8d52660ca362a7a12e14c2c3f890458f9cb51544580ced922a"} Oct 30 19:32:27.047808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.047780 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:27.110515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.110480 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\") pod \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " Oct 30 19:32:27.110738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.110543 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\" (UniqueName: \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\") pod \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " Oct 30 19:32:27.110738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.110672 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\") pod \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\" (UID: \"92672c4e-204a-4e2e-a9ad-51f08dff8f80\") " Oct 30 19:32:27.110976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.110946 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca" pod "92672c4e-204a-4e2e-a9ad-51f08dff8f80" (UID: "92672c4e-204a-4e2e-a9ad-51f08dff8f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:27.111099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.111081 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.112823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.112789 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn" pod "92672c4e-204a-4e2e-a9ad-51f08dff8f80" (UID: "92672c4e-204a-4e2e-a9ad-51f08dff8f80"). InnerVolumeSpecName "kube-api-access-grbcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:27.112928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.112903 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca" pod "92672c4e-204a-4e2e-a9ad-51f08dff8f80" (UID: "92672c4e-204a-4e2e-a9ad-51f08dff8f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:27.212574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.212534 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbcn\" (UniqueName: \"kubernetes.io/projected/92672c4e-204a-4e2e-a9ad-51f08dff8f80-kube-api-access-grbcn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.212574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.212570 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92672c4e-204a-4e2e-a9ad-51f08dff8f80-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.886718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.886682 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" event={"ID":"92672c4e-204a-4e2e-a9ad-51f08dff8f80","Type":"ContainerDied","Data":"a31d7d99d39d8cb19fbbc80f7ab0ab158f6f99fb7e42d1158053831d45242407"} Oct 30 19:32:27.886718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.886708 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc" Oct 30 19:32:27.886936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:27.886739 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31d7d99d39d8cb19fbbc80f7ab0ab158f6f99fb7e42d1158053831d45242407" Oct 30 19:32:34.868850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:34.868820 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:32:39.985786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.985748 4789 generic.go:356] "Generic (PLEG): container finished" podID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerID="77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef" exitCode=0 Oct 30 19:32:39.986289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.985826 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerDied","Data":"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef"} Oct 30 19:32:39.986342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.986312 4789 scope.go:122] "RemoveContainer" containerID="77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef" Oct 30 19:32:39.988156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.988120 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:39.988280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.988232 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:39.988328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:39.988282 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:32:40.996234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:40.995746 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerStarted","Data":"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5"} Oct 30 19:32:41.249863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.249775 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:32:41.250076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.250059 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:32:41.250121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.250100 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:32:41.822734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.822697 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8"] Oct 30 19:32:41.823318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.823304 4789 cpu_manager.go:414] "containerMap: removing container" podUID="92672c4e-204a-4e2e-a9ad-51f08dff8f80" containerName="revoke-keys" Oct 30 19:32:41.823369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.823321 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92672c4e-204a-4e2e-a9ad-51f08dff8f80" containerName="revoke-keys" Oct 30 19:32:41.823472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.823461 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="92672c4e-204a-4e2e-a9ad-51f08dff8f80" containerName="revoke-keys" Oct 30 19:32:41.825873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.825854 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.829583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.829555 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp76p\" (UniqueName: \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.829707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.829601 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.829762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.829705 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.872474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.872452 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8"] Oct 30 19:32:41.875723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.875700 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:41.875869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.875750 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:32:41.875988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.875963 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:32:41.915405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.915364 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:32:41.930298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.930262 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp76p\" (UniqueName: \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.930468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.930303 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.930468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.930342 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.931180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.931153 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.932849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.932819 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:41.988710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:41.988674 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp76p\" (UniqueName: \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\") pod \"maas-api-revoke-sub-6ff307ec-mjwx8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:42.139088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:42.139012 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:42.250172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:42.250097 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:32:42.250368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:42.250165 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:32:42.518989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:42.518948 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8"] Oct 30 19:32:42.522173 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:32:42.522140 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e2dd7f_73cc_498f_9136_86a16f8569d8.slice/crio-7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c WatchSource:0}: Error finding container 7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c: Status 404 returned error can't find the container with id 7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c Oct 30 19:32:43.030642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:43.030609 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" event={"ID":"82e2dd7f-73cc-498f-9136-86a16f8569d8","Type":"ContainerStarted","Data":"e6ab908acbb9c8074508dceaaf0081b8fa4e96ad01d5169f8a6aaaeea389e402"} Oct 30 19:32:43.030797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:43.030661 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" event={"ID":"82e2dd7f-73cc-498f-9136-86a16f8569d8","Type":"ContainerStarted","Data":"7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c"} Oct 30 19:32:43.141042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:43.140987 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" podStartSLOduration=2.140975656 podStartE2EDuration="2.140975656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:43.140436783 +0000 UTC m=+2799.642347269" watchObservedRunningTime="2026-10-30 19:32:43.140975656 +0000 UTC m=+2799.642886142" Oct 30 19:32:43.250244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:43.250187 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:32:43.250413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:43.250262 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:32:44.039606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:44.039571 4789 generic.go:356] "Generic (PLEG): container finished" podID="82e2dd7f-73cc-498f-9136-86a16f8569d8" containerID="e6ab908acbb9c8074508dceaaf0081b8fa4e96ad01d5169f8a6aaaeea389e402" exitCode=0 Oct 30 19:32:44.039799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:44.039643 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" event={"ID":"82e2dd7f-73cc-498f-9136-86a16f8569d8","Type":"ContainerDied","Data":"e6ab908acbb9c8074508dceaaf0081b8fa4e96ad01d5169f8a6aaaeea389e402"} Oct 30 19:32:44.250583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:44.250548 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:32:44.251036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:44.250608 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:32:45.243034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.242997 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:45.250094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.250072 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 30 19:32:45.250197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.250113 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 30 19:32:45.260544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.260518 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\") pod \"82e2dd7f-73cc-498f-9136-86a16f8569d8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " Oct 30 19:32:45.260817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.260664 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\") pod \"82e2dd7f-73cc-498f-9136-86a16f8569d8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " Oct 30 19:32:45.260817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.260703 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\" (UniqueName: \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\") pod \"82e2dd7f-73cc-498f-9136-86a16f8569d8\" (UID: \"82e2dd7f-73cc-498f-9136-86a16f8569d8\") " Oct 30 19:32:45.261022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.261002 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca" pod "82e2dd7f-73cc-498f-9136-86a16f8569d8" (UID: "82e2dd7f-73cc-498f-9136-86a16f8569d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:45.262538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.262518 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca" pod "82e2dd7f-73cc-498f-9136-86a16f8569d8" (UID: "82e2dd7f-73cc-498f-9136-86a16f8569d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:45.262756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.262743 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p" pod "82e2dd7f-73cc-498f-9136-86a16f8569d8" (UID: "82e2dd7f-73cc-498f-9136-86a16f8569d8"). InnerVolumeSpecName "kube-api-access-lp76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:45.362110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.362032 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:45.362110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.362066 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e2dd7f-73cc-498f-9136-86a16f8569d8-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:45.362110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:45.362079 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp76p\" (UniqueName: \"kubernetes.io/projected/82e2dd7f-73cc-498f-9136-86a16f8569d8-kube-api-access-lp76p\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:46.052590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.052548 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" event={"ID":"82e2dd7f-73cc-498f-9136-86a16f8569d8","Type":"ContainerDied","Data":"7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c"} Oct 30 19:32:46.052810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.052614 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ed15f2c11830b9983615178fde7a64b33cb060fb1f49e2d0c841fbca961878c" Oct 30 19:32:46.052810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.052680 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8" Oct 30 19:32:46.254399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.254370 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:32:46.254772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.254752 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:32:46.255416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:46.255400 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:32:48.196124 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:32:48.196089 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:32:54.139279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.139248 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:32:54.139715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.139490 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" podUID="4eb356a8-a99a-4b18-b5c1-47212269ea45" containerName="payload-processing" containerID="cri-o://33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915" gracePeriod=30 Oct 30 19:32:54.248420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.248386 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:32:54.248599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.248576 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" podUID="884841f5-5632-4f49-a964-96c7f5e6ae0e" containerName="payload-pre-processing" containerID="cri-o://84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b" gracePeriod=30 Oct 30 19:32:54.423984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.423961 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:32:54.456609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.456566 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\" (UniqueName: \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\") pod \"4eb356a8-a99a-4b18-b5c1-47212269ea45\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " Oct 30 19:32:54.456836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.456816 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\") pod \"4eb356a8-a99a-4b18-b5c1-47212269ea45\" (UID: \"4eb356a8-a99a-4b18-b5c1-47212269ea45\") " Oct 30 19:32:54.457312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.457288 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume" pod "4eb356a8-a99a-4b18-b5c1-47212269ea45" (UID: "4eb356a8-a99a-4b18-b5c1-47212269ea45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:54.460026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.459998 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn" pod "4eb356a8-a99a-4b18-b5c1-47212269ea45" (UID: "4eb356a8-a99a-4b18-b5c1-47212269ea45"). InnerVolumeSpecName "kube-api-access-bdcmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:54.513557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.513532 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:32:54.557667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.557637 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\") pod \"884841f5-5632-4f49-a964-96c7f5e6ae0e\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " Oct 30 19:32:54.557886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.557862 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\" (UniqueName: \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\") pod \"884841f5-5632-4f49-a964-96c7f5e6ae0e\" (UID: \"884841f5-5632-4f49-a964-96c7f5e6ae0e\") " Oct 30 19:32:54.558017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.557997 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume" pod "884841f5-5632-4f49-a964-96c7f5e6ae0e" (UID: "884841f5-5632-4f49-a964-96c7f5e6ae0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:54.558241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.558194 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb356a8-a99a-4b18-b5c1-47212269ea45-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.558241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.558228 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdcmn\" (UniqueName: \"kubernetes.io/projected/4eb356a8-a99a-4b18-b5c1-47212269ea45-kube-api-access-bdcmn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.558241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.558238 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884841f5-5632-4f49-a964-96c7f5e6ae0e-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.559938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.559911 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf" pod "884841f5-5632-4f49-a964-96c7f5e6ae0e" (UID: "884841f5-5632-4f49-a964-96c7f5e6ae0e"). InnerVolumeSpecName "kube-api-access-2rdlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:54.676908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:54.676826 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdlf\" (UniqueName: \"kubernetes.io/projected/884841f5-5632-4f49-a964-96c7f5e6ae0e-kube-api-access-2rdlf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:55.108000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.107962 4789 generic.go:356] "Generic (PLEG): container finished" podID="4eb356a8-a99a-4b18-b5c1-47212269ea45" containerID="33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915" exitCode=0 Oct 30 19:32:55.108188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.108009 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" Oct 30 19:32:55.108188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.108041 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" event={"ID":"4eb356a8-a99a-4b18-b5c1-47212269ea45","Type":"ContainerDied","Data":"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915"} Oct 30 19:32:55.108188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.108100 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5" event={"ID":"4eb356a8-a99a-4b18-b5c1-47212269ea45","Type":"ContainerDied","Data":"1fdb889b664a7e7509def2b69efa7f390cecda8bf3a205a6f1a82107b1b31c05"} Oct 30 19:32:55.108188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.108127 4789 scope.go:122] "RemoveContainer" containerID="33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915" Oct 30 19:32:55.110356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.110239 4789 generic.go:356] "Generic (PLEG): container finished" podID="884841f5-5632-4f49-a964-96c7f5e6ae0e" containerID="84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b" exitCode=0 Oct 30 19:32:55.110356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.110278 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" event={"ID":"884841f5-5632-4f49-a964-96c7f5e6ae0e","Type":"ContainerDied","Data":"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b"} Oct 30 19:32:55.110356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.110302 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" Oct 30 19:32:55.110356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.110342 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq" event={"ID":"884841f5-5632-4f49-a964-96c7f5e6ae0e","Type":"ContainerDied","Data":"730612461ceb0e290d3f306574bbb71080f937bedd0baf3407ca98d4b7bc6af8"} Oct 30 19:32:55.120636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.120613 4789 scope.go:122] "RemoveContainer" containerID="33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915" Oct 30 19:32:55.121038 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:32:55.120954 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915\": container with ID starting with 33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915 not found: ID does not exist" containerID="33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915" Oct 30 19:32:55.121038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.120986 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915"} err="failed to get container status \"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915\": rpc error: code = NotFound desc = could not find container \"33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915\": container with ID starting with 33864d6103167d3834013c80478ae07e5e509f0b907fabd3dea68ed8f2577915 not found: ID does not exist" Oct 30 19:32:55.121038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.121011 4789 scope.go:122] "RemoveContainer" containerID="84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b" Oct 30 19:32:55.130138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.130118 4789 scope.go:122] "RemoveContainer" containerID="84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b" Oct 30 19:32:55.130473 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:32:55.130450 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b\": container with ID starting with 84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b not found: ID does not exist" containerID="84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b" Oct 30 19:32:55.130542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.130486 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b"} err="failed to get container status \"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b\": rpc error: code = NotFound desc = could not find container \"84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b\": container with ID starting with 84ce6828f38d801320484331d4c7c3ac7d2ee78555e18286bf40396fd38d498b not found: ID does not exist" Oct 30 19:32:55.266670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.266631 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:32:55.272740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.272711 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a790b-5794d8d8b6-fgsq5"] Oct 30 19:32:55.375753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.375659 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:32:55.386188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.386154 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a790b-7f5b744bdc-828bq"] Oct 30 19:32:55.668567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.668492 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb356a8-a99a-4b18-b5c1-47212269ea45" path="/var/lib/kubelet/pods/4eb356a8-a99a-4b18-b5c1-47212269ea45/volumes" Oct 30 19:32:55.668898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:55.668885 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884841f5-5632-4f49-a964-96c7f5e6ae0e" path="/var/lib/kubelet/pods/884841f5-5632-4f49-a964-96c7f5e6ae0e/volumes" Oct 30 19:32:56.598165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:56.598131 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn"] Oct 30 19:32:56.644237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:56.644182 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7a790b-xvwdn"] Oct 30 19:32:57.667466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:57.667420 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c777293-6889-4b02-b197-82be02f56a02" path="/var/lib/kubelet/pods/6c777293-6889-4b02-b197-82be02f56a02/volumes" Oct 30 19:32:59.596408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.596373 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:32:59.597021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597002 4789 cpu_manager.go:414] "containerMap: removing container" podUID="884841f5-5632-4f49-a964-96c7f5e6ae0e" containerName="payload-pre-processing" Oct 30 19:32:59.597077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597024 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884841f5-5632-4f49-a964-96c7f5e6ae0e" containerName="payload-pre-processing" Oct 30 19:32:59.597077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597053 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4eb356a8-a99a-4b18-b5c1-47212269ea45" containerName="payload-processing" Oct 30 19:32:59.597077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597059 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb356a8-a99a-4b18-b5c1-47212269ea45" containerName="payload-processing" Oct 30 19:32:59.597184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597087 4789 cpu_manager.go:414] "containerMap: removing container" podUID="82e2dd7f-73cc-498f-9136-86a16f8569d8" containerName="revoke-keys" Oct 30 19:32:59.597184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597092 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e2dd7f-73cc-498f-9136-86a16f8569d8" containerName="revoke-keys" Oct 30 19:32:59.597184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597164 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb356a8-a99a-4b18-b5c1-47212269ea45" containerName="payload-processing" Oct 30 19:32:59.597184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597178 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e2dd7f-73cc-498f-9136-86a16f8569d8" containerName="revoke-keys" Oct 30 19:32:59.597353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.597186 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="884841f5-5632-4f49-a964-96c7f5e6ae0e" containerName="payload-pre-processing" Oct 30 19:32:59.600241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.600226 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.695442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.695399 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-83ac32\"" Oct 30 19:32:59.695703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.695474 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-83ac32-dockercfg-tzmww\"" Oct 30 19:32:59.702337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.702305 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:32:59.733285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.733247 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.733470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.733384 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68dp\" (UniqueName: \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.763495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.763465 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:32:59.766180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.766164 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.822124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.822098 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:32:59.834589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.834561 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2489x\" (UniqueName: \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.834738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.834713 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.834815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.834764 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.834869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.834831 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68dp\" (UniqueName: \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.835417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.835393 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.892882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.892807 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68dp\" (UniqueName: \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\") pod \"payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.912575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.912544 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:32:59.936032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.935998 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.936163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.936097 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2489x\" (UniqueName: \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.936717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.936697 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:32:59.996374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:32:59.996345 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2489x\" (UniqueName: \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\") pod \"payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:33:00.078521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:00.078487 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:33:00.285379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:00.285351 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:33:00.288178 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:00.288151 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bad0e8d_e978_49e7_a823_b9ab84622b17.slice/crio-5bd003163eeeab2c1c070dcf38938ae90385bf3395d5d106ee9e8467fdc3b3d1 WatchSource:0}: Error finding container 5bd003163eeeab2c1c070dcf38938ae90385bf3395d5d106ee9e8467fdc3b3d1: Status 404 returned error can't find the container with id 5bd003163eeeab2c1c070dcf38938ae90385bf3395d5d106ee9e8467fdc3b3d1 Oct 30 19:33:00.449027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:00.448993 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:33:00.455291 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:00.455263 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c6b24e_40b8_4f9d_9315_dbb72ab7a043.slice/crio-cd9c305104a0c61b0a2f10934814b011ebbe4a95996686baadc5cb0387ac8cd0 WatchSource:0}: Error finding container cd9c305104a0c61b0a2f10934814b011ebbe4a95996686baadc5cb0387ac8cd0: Status 404 returned error can't find the container with id cd9c305104a0c61b0a2f10934814b011ebbe4a95996686baadc5cb0387ac8cd0 Oct 30 19:33:01.158044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.158002 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" event={"ID":"8bad0e8d-e978-49e7-a823-b9ab84622b17","Type":"ContainerStarted","Data":"0c07e5ba50334a35fa09aac494338a4baecda909c64a2cc3bcbe7cfdb6371c1d"} Oct 30 19:33:01.158553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.158060 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" event={"ID":"8bad0e8d-e978-49e7-a823-b9ab84622b17","Type":"ContainerStarted","Data":"5bd003163eeeab2c1c070dcf38938ae90385bf3395d5d106ee9e8467fdc3b3d1"} Oct 30 19:33:01.158553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.158083 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:33:01.159791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.159756 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" event={"ID":"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043","Type":"ContainerStarted","Data":"baeb4d1e94d136a08548d006c9a06e31ec9c14b5cb0c19da7ac8abed29537145"} Oct 30 19:33:01.159917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.159813 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" event={"ID":"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043","Type":"ContainerStarted","Data":"cd9c305104a0c61b0a2f10934814b011ebbe4a95996686baadc5cb0387ac8cd0"} Oct 30 19:33:01.159917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.159854 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:33:01.272064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.271928 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" podStartSLOduration=2.27191136 podStartE2EDuration="2.27191136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:01.270917515 +0000 UTC m=+2817.772828006" watchObservedRunningTime="2026-10-30 19:33:01.27191136 +0000 UTC m=+2817.773821845" Oct 30 19:33:01.382034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.381984 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" podStartSLOduration=2.381965752 podStartE2EDuration="2.381965752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:01.379661039 +0000 UTC m=+2817.881571525" watchObservedRunningTime="2026-10-30 19:33:01.381965752 +0000 UTC m=+2817.883876269" Oct 30 19:33:01.993638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:01.993603 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:33:06.255117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:06.255077 4789 scope.go:122] "RemoveContainer" containerID="305d7986b4676b5c60be9e63da3b5ab128de4b294f3acc42cdc3a54a8c3419f8" Oct 30 19:33:08.411932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.411890 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz"] Oct 30 19:33:08.417545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.417521 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.471414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.471377 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz"] Oct 30 19:33:08.513820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.513780 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:33:08.513820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.513807 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:33:08.513820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.513782 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:33:08.514106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.513953 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:33:08.534852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.534821 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.534852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.534855 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.535093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.535001 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46wz\" (UniqueName: \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.636000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.635960 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.636000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.636004 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.636203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.636129 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46wz\" (UniqueName: \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.636732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.636714 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.638403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.638386 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.694898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.694869 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46wz\" (UniqueName: \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\") pod \"maas-api-revoke-sub-290921f6-wxztz\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:08.730792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:08.730762 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:09.140485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:09.140336 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz"] Oct 30 19:33:09.143193 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:09.143161 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd470778a_ab02_4b6d_a865_51eb7c27aadf.slice/crio-468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613 WatchSource:0}: Error finding container 468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613: Status 404 returned error can't find the container with id 468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613 Oct 30 19:33:09.213892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:09.213854 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" event={"ID":"d470778a-ab02-4b6d-a865-51eb7c27aadf","Type":"ContainerStarted","Data":"468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613"} Oct 30 19:33:10.223078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:10.222984 4789 generic.go:356] "Generic (PLEG): container finished" podID="d470778a-ab02-4b6d-a865-51eb7c27aadf" containerID="e8253a1e2a85bffb778a5aaa3db1d1c26e7b8ee7474482f08fad106a1369df5b" exitCode=0 Oct 30 19:33:10.223500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:10.223069 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" event={"ID":"d470778a-ab02-4b6d-a865-51eb7c27aadf","Type":"ContainerDied","Data":"e8253a1e2a85bffb778a5aaa3db1d1c26e7b8ee7474482f08fad106a1369df5b"} Oct 30 19:33:11.383945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.383913 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:11.570133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.570039 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\") pod \"d470778a-ab02-4b6d-a865-51eb7c27aadf\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " Oct 30 19:33:11.570133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.570088 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\" (UniqueName: \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\") pod \"d470778a-ab02-4b6d-a865-51eb7c27aadf\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " Oct 30 19:33:11.570421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.570242 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\") pod \"d470778a-ab02-4b6d-a865-51eb7c27aadf\" (UID: \"d470778a-ab02-4b6d-a865-51eb7c27aadf\") " Oct 30 19:33:11.570567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.570544 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca" pod "d470778a-ab02-4b6d-a865-51eb7c27aadf" (UID: "d470778a-ab02-4b6d-a865-51eb7c27aadf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:11.572275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.572252 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca" pod "d470778a-ab02-4b6d-a865-51eb7c27aadf" (UID: "d470778a-ab02-4b6d-a865-51eb7c27aadf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:11.572369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.572347 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz" pod "d470778a-ab02-4b6d-a865-51eb7c27aadf" (UID: "d470778a-ab02-4b6d-a865-51eb7c27aadf"). InnerVolumeSpecName "kube-api-access-q46wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:11.671786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.671750 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:11.671786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.671787 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d470778a-ab02-4b6d-a865-51eb7c27aadf-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:11.671786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:11.671803 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46wz\" (UniqueName: \"kubernetes.io/projected/d470778a-ab02-4b6d-a865-51eb7c27aadf-kube-api-access-q46wz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:12.169297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:12.169267 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:33:12.169504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:12.169344 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:33:12.239041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:12.239008 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" Oct 30 19:33:12.239041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:12.239020 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz" event={"ID":"d470778a-ab02-4b6d-a865-51eb7c27aadf","Type":"ContainerDied","Data":"468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613"} Oct 30 19:33:12.239270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:12.239066 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468fa94ba4b970b05583dd625c1477efec0484adb3118db9d92de5d3f0cc2613" Oct 30 19:33:17.014697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.014635 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerName="main" containerID="cri-o://4125b2b4e5b232bf12de95aee2894da5f3dada711a63f911698a451b20ca3247" gracePeriod=45 Oct 30 19:33:17.270693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.270664 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerID="4125b2b4e5b232bf12de95aee2894da5f3dada711a63f911698a451b20ca3247" exitCode=0 Oct 30 19:33:17.270828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.270731 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" event={"ID":"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2","Type":"ContainerDied","Data":"4125b2b4e5b232bf12de95aee2894da5f3dada711a63f911698a451b20ca3247"} Oct 30 19:33:17.292817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.292794 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:33:17.328110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328070 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\" (UniqueName: \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328130 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328166 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328251 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328291 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\") pod \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\" (UID: \"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2\") " Oct 30 19:33:17.328996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.328963 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:17.329142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.329046 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:17.329514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.329488 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:17.331354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.331313 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:17.331479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.331376 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "kube-api-access-lk6b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:17.332238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.332177 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs" pod "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" (UID: "dd3cf2f5-f940-4ad7-b22a-7dffa35983d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:17.430477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430439 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk6b8\" (UniqueName: \"kubernetes.io/projected/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-kube-api-access-lk6b8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:17.430477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430476 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:17.430765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430493 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:17.430765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430509 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:17.430765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430524 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:17.430765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:17.430537 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.196238 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:33:18.196187 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:33:18.278610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.278581 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" Oct 30 19:33:18.278787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.278625 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" event={"ID":"dd3cf2f5-f940-4ad7-b22a-7dffa35983d2","Type":"ContainerDied","Data":"14e19ccc15eba63372c6111bab0002d5f795b9438184288d9d18b3ec68330b7a"} Oct 30 19:33:18.278787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.278671 4789 scope.go:122] "RemoveContainer" containerID="4125b2b4e5b232bf12de95aee2894da5f3dada711a63f911698a451b20ca3247" Oct 30 19:33:18.284277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.284246 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:18.439905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.439868 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:33:18.443100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:18.443075 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7a790b/ipp-route-7a790b-kserve-788b76f67c-crszt"] Oct 30 19:33:19.670195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:19.670164 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" path="/var/lib/kubelet/pods/dd3cf2f5-f940-4ad7-b22a-7dffa35983d2/volumes" Oct 30 19:33:20.506843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.506799 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc"] Oct 30 19:33:20.507392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507371 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d470778a-ab02-4b6d-a865-51eb7c27aadf" containerName="revoke-keys" Oct 30 19:33:20.507522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507394 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d470778a-ab02-4b6d-a865-51eb7c27aadf" containerName="revoke-keys" Oct 30 19:33:20.507522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507429 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerName="main" Oct 30 19:33:20.507522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507444 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerName="main" Oct 30 19:33:20.507691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507608 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d470778a-ab02-4b6d-a865-51eb7c27aadf" containerName="revoke-keys" Oct 30 19:33:20.507691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.507627 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3cf2f5-f940-4ad7-b22a-7dffa35983d2" containerName="main" Oct 30 19:33:20.521902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.521844 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.561402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.561370 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.561571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.561428 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.561571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.561499 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h282\" (UniqueName: \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.568136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.568103 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc"] Oct 30 19:33:20.610229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.610177 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:33:20.610229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.610188 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:33:20.610229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.610194 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:33:20.617040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.617015 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:33:20.662555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.662527 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.662716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.662565 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.662716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.662590 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h282\" (UniqueName: \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.663120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.663099 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.664967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.664942 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.720569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.720537 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h282\" (UniqueName: \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\") pod \"maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:20.835162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:20.835083 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:21.209866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:21.209830 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc"] Oct 30 19:33:21.212580 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:21.212529 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c23645_7496_47ed_bc0a_74696e82b80c.slice/crio-9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43 WatchSource:0}: Error finding container 9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43: Status 404 returned error can't find the container with id 9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43 Oct 30 19:33:21.301539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:21.301502 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" event={"ID":"49c23645-7496-47ed-bc0a-74696e82b80c","Type":"ContainerStarted","Data":"9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43"} Oct 30 19:33:22.309482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:22.309443 4789 generic.go:356] "Generic (PLEG): container finished" podID="49c23645-7496-47ed-bc0a-74696e82b80c" containerID="592d30070d420f59dfac5366e9451b88d974005bf1a3f87ac9a8e1572de3022c" exitCode=0 Oct 30 19:33:22.309972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:22.309526 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" event={"ID":"49c23645-7496-47ed-bc0a-74696e82b80c","Type":"ContainerDied","Data":"592d30070d420f59dfac5366e9451b88d974005bf1a3f87ac9a8e1572de3022c"} Oct 30 19:33:23.471087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.471063 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:23.489992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.489965 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\") pod \"49c23645-7496-47ed-bc0a-74696e82b80c\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " Oct 30 19:33:23.490135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.490014 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\") pod \"49c23645-7496-47ed-bc0a-74696e82b80c\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " Oct 30 19:33:23.490205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.490174 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\" (UniqueName: \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\") pod \"49c23645-7496-47ed-bc0a-74696e82b80c\" (UID: \"49c23645-7496-47ed-bc0a-74696e82b80c\") " Oct 30 19:33:23.490384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.490361 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca" pod "49c23645-7496-47ed-bc0a-74696e82b80c" (UID: "49c23645-7496-47ed-bc0a-74696e82b80c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:23.490619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.490601 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:23.492102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.492073 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282" pod "49c23645-7496-47ed-bc0a-74696e82b80c" (UID: "49c23645-7496-47ed-bc0a-74696e82b80c"). InnerVolumeSpecName "kube-api-access-4h282". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:23.492317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.492294 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca" pod "49c23645-7496-47ed-bc0a-74696e82b80c" (UID: "49c23645-7496-47ed-bc0a-74696e82b80c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:23.591992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.591923 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h282\" (UniqueName: \"kubernetes.io/projected/49c23645-7496-47ed-bc0a-74696e82b80c-kube-api-access-4h282\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:23.591992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:23.591950 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49c23645-7496-47ed-bc0a-74696e82b80c-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:24.324233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.324176 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" Oct 30 19:33:24.324456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.324176 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc" event={"ID":"49c23645-7496-47ed-bc0a-74696e82b80c","Type":"ContainerDied","Data":"9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43"} Oct 30 19:33:24.324456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.324351 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da90fe6f5f04f8e09c7d378515bd575e097fedf0c1f7a150c8cdbc550fb3d43" Oct 30 19:33:24.556073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.556041 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9"] Oct 30 19:33:24.556600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.556585 4789 cpu_manager.go:414] "containerMap: removing container" podUID="49c23645-7496-47ed-bc0a-74696e82b80c" containerName="revoke-keys" Oct 30 19:33:24.556659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.556602 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c23645-7496-47ed-bc0a-74696e82b80c" containerName="revoke-keys" Oct 30 19:33:24.556723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.556712 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c23645-7496-47ed-bc0a-74696e82b80c" containerName="revoke-keys" Oct 30 19:33:24.560201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.560185 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.602153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.602058 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsp6p\" (UniqueName: \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.602153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.602121 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.602381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.602249 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.613084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.613048 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:33:24.613248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.613223 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:33:24.659506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.659471 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:33:24.659648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.659529 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:33:24.663891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.663869 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9"] Oct 30 19:33:24.703375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.703340 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsp6p\" (UniqueName: \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.703537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.703474 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.703580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.703535 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.704106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.704086 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.705823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.705801 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.761433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.761404 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsp6p\" (UniqueName: \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\") pod \"maas-api-revoke-sub-a076be9c-8fdc9\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:24.873479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:24.873404 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:25.249273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.249249 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9"] Oct 30 19:33:25.251977 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:25.251949 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5127e126_a0c4_41df_bc8e_9d86c0e581ab.slice/crio-477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce WatchSource:0}: Error finding container 477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce: Status 404 returned error can't find the container with id 477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce Oct 30 19:33:25.297475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.297452 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7"] Oct 30 19:33:25.302459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.302443 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.332306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.332275 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" event={"ID":"5127e126-a0c4-41df-bc8e-9d86c0e581ab","Type":"ContainerStarted","Data":"477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce"} Oct 30 19:33:25.356263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.356204 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7"] Oct 30 19:33:25.412033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.412005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.412263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.412152 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.412263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.412184 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.513463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.513438 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.513578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.513473 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.513578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.513532 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.514119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.514100 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.515791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.515772 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.571124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.571096 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\") pod \"maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.615256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.615234 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:25.989785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:25.989752 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7"] Oct 30 19:33:25.991981 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:25.991951 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80f4d93_90e0_4279_90bb_e06ecd5a9ccd.slice/crio-a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf WatchSource:0}: Error finding container a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf: Status 404 returned error can't find the container with id a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf Oct 30 19:33:26.341041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:26.341004 4789 generic.go:356] "Generic (PLEG): container finished" podID="5127e126-a0c4-41df-bc8e-9d86c0e581ab" containerID="96e92f04f3ab8d93717ec10f465490ee2f0c076e254f131fd279ecff245d146b" exitCode=0 Oct 30 19:33:26.341234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:26.341092 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" event={"ID":"5127e126-a0c4-41df-bc8e-9d86c0e581ab","Type":"ContainerDied","Data":"96e92f04f3ab8d93717ec10f465490ee2f0c076e254f131fd279ecff245d146b"} Oct 30 19:33:26.342845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:26.342822 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" event={"ID":"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd","Type":"ContainerStarted","Data":"21e87c16e443e9029c3781487348eb1a3a814bfa50a3d2a67ad2404a29bd2e12"} Oct 30 19:33:26.342954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:26.342864 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" event={"ID":"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd","Type":"ContainerStarted","Data":"a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf"} Oct 30 19:33:26.563696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:26.563589 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" podStartSLOduration=1.563569972 podStartE2EDuration="1.563569972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:26.559951692 +0000 UTC m=+2843.061862177" watchObservedRunningTime="2026-10-30 19:33:26.563569972 +0000 UTC m=+2843.065480452" Oct 30 19:33:27.350248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.350146 4789 generic.go:356] "Generic (PLEG): container finished" podID="f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" containerID="21e87c16e443e9029c3781487348eb1a3a814bfa50a3d2a67ad2404a29bd2e12" exitCode=0 Oct 30 19:33:27.350248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.350204 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" event={"ID":"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd","Type":"ContainerDied","Data":"21e87c16e443e9029c3781487348eb1a3a814bfa50a3d2a67ad2404a29bd2e12"} Oct 30 19:33:27.507673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.507644 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:27.642306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.642201 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\" (UniqueName: \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\") pod \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " Oct 30 19:33:27.642507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.642396 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\") pod \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " Oct 30 19:33:27.642507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.642450 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\") pod \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\" (UID: \"5127e126-a0c4-41df-bc8e-9d86c0e581ab\") " Oct 30 19:33:27.642924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.642889 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca" pod "5127e126-a0c4-41df-bc8e-9d86c0e581ab" (UID: "5127e126-a0c4-41df-bc8e-9d86c0e581ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:27.644414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.644391 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca" pod "5127e126-a0c4-41df-bc8e-9d86c0e581ab" (UID: "5127e126-a0c4-41df-bc8e-9d86c0e581ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:27.644510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.644424 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p" pod "5127e126-a0c4-41df-bc8e-9d86c0e581ab" (UID: "5127e126-a0c4-41df-bc8e-9d86c0e581ab"). InnerVolumeSpecName "kube-api-access-wsp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.743661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.743628 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.743661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.743656 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5127e126-a0c4-41df-bc8e-9d86c0e581ab-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.743856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:27.743668 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsp6p\" (UniqueName: \"kubernetes.io/projected/5127e126-a0c4-41df-bc8e-9d86c0e581ab-kube-api-access-wsp6p\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:28.358405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.358369 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" event={"ID":"5127e126-a0c4-41df-bc8e-9d86c0e581ab","Type":"ContainerDied","Data":"477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce"} Oct 30 19:33:28.358405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.358392 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9" Oct 30 19:33:28.358855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.358423 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477ee492d529a3a4263eee17629b72fb9a797fb81eb845af21dfc82dcb68f5ce" Oct 30 19:33:28.504310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.504273 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:28.552435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.552405 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\") pod \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " Oct 30 19:33:28.552605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.552470 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\") pod \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " Oct 30 19:33:28.552605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.552571 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\") pod \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\" (UID: \"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd\") " Oct 30 19:33:28.552902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.552875 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca" pod "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" (UID: "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:28.554574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.554545 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca" pod "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" (UID: "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:28.554658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.554600 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4" pod "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" (UID: "f80f4d93-90e0-4279-90bb-e06ecd5a9ccd"). InnerVolumeSpecName "kube-api-access-56pw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:28.654233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.654136 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:28.654233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.654165 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56pw4\" (UniqueName: \"kubernetes.io/projected/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-kube-api-access-56pw4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:28.654233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:28.654179 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.366014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:29.365981 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" event={"ID":"f80f4d93-90e0-4279-90bb-e06ecd5a9ccd","Type":"ContainerDied","Data":"a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf"} Oct 30 19:33:29.366014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:29.365998 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7" Oct 30 19:33:29.366505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:29.366030 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a801f1c46cd2258b3db4c7b338343fbccb95d654d5812f2c79b0183bea0d2ddf" Oct 30 19:33:43.808435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:43.808395 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc"] Oct 30 19:33:43.853015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:43.852982 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f9a20d-pf4rc"] Oct 30 19:33:45.667901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:45.667871 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c23645-7496-47ed-bc0a-74696e82b80c" path="/var/lib/kubelet/pods/49c23645-7496-47ed-bc0a-74696e82b80c/volumes" Oct 30 19:33:48.196255 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:33:48.196203 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:33:52.106409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.106360 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:33:52.107018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.106997 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" containerName="revoke-keys" Oct 30 19:33:52.107077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.107021 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" containerName="revoke-keys" Oct 30 19:33:52.107077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.107056 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5127e126-a0c4-41df-bc8e-9d86c0e581ab" containerName="revoke-keys" Oct 30 19:33:52.107077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.107062 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5127e126-a0c4-41df-bc8e-9d86c0e581ab" containerName="revoke-keys" Oct 30 19:33:52.107194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.107149 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" containerName="revoke-keys" Oct 30 19:33:52.107194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.107161 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5127e126-a0c4-41df-bc8e-9d86c0e581ab" containerName="revoke-keys" Oct 30 19:33:52.111902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.111882 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.161279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.161255 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-330025-openshift-default-dockercfg-9qb6q\"" Oct 30 19:33:52.210414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210386 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:33:52.210825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210803 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.210948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210835 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.210948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210866 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.210948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210911 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.210948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210932 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.211179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.210996 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.211179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.211050 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cx2r\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.211179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.211083 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.211179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.211113 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312064 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312133 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312162 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312196 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312268 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312293 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312360 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312417 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cx2r\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312450 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312488 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312761 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.312940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312790 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.313051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.312953 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.313051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.313032 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.315024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.315002 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.315237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.315200 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.370104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.370045 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.370205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.370110 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cx2r\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.425992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.425962 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:52.852999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.852969 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:33:52.854370 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:33:52.854345 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1949abeb_e731_4bb4_8351_a9e2e80ac9fc.slice/crio-ff8956dc81350d6ab66ae54d10da5a2745b65c67120880a5d98fe99b44cefff8 WatchSource:0}: Error finding container ff8956dc81350d6ab66ae54d10da5a2745b65c67120880a5d98fe99b44cefff8: Status 404 returned error can't find the container with id ff8956dc81350d6ab66ae54d10da5a2745b65c67120880a5d98fe99b44cefff8 Oct 30 19:33:52.856697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.856668 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:52.856779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.856729 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:52.856779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:52.856765 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:33:53.528392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:53.528350 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" event={"ID":"1949abeb-e731-4bb4-8351-a9e2e80ac9fc","Type":"ContainerStarted","Data":"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6"} Oct 30 19:33:53.528867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:53.528434 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" event={"ID":"1949abeb-e731-4bb4-8351-a9e2e80ac9fc","Type":"ContainerStarted","Data":"ff8956dc81350d6ab66ae54d10da5a2745b65c67120880a5d98fe99b44cefff8"} Oct 30 19:33:53.699676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:53.692471 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podStartSLOduration=1.692452394 podStartE2EDuration="1.692452394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:53.689166995 +0000 UTC m=+2870.191077481" watchObservedRunningTime="2026-10-30 19:33:53.692452394 +0000 UTC m=+2870.194362880" Oct 30 19:33:54.426577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:54.426538 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:33:54.428223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:54.428174 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:54.428376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:54.428261 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:55.426840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:55.426801 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:55.427346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:55.426870 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:56.427198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:56.427154 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:56.427638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:56.427275 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:57.427787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:57.427264 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:57.427787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:57.427334 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:58.426691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:58.426659 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:58.426872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:58.426712 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:59.348964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:59.348924 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7"] Oct 30 19:33:59.394530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:59.394498 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-441836-lrmh7"] Oct 30 19:33:59.427294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:59.427265 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 30 19:33:59.427411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:59.427322 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 30 19:33:59.667428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:33:59.667356 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80f4d93-90e0-4279-90bb-e06ecd5a9ccd" path="/var/lib/kubelet/pods/f80f4d93-90e0-4279-90bb-e06ecd5a9ccd/volumes" Oct 30 19:34:00.430683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.430655 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:34:00.431110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.431087 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:34:00.431877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.431862 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:34:00.794767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.794734 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94"] Oct 30 19:34:00.798528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.798509 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:00.850565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.850543 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:00.850795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.850770 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:34:00.850959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.850832 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:00.850959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.850646 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:00.854639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.854616 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94"] Oct 30 19:34:00.905889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.905865 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9v9v\" (UniqueName: \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:00.906007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.905969 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:00.906007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:00.906002 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.006958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.006932 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9v9v\" (UniqueName: \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.007127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.006988 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.007127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.007039 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.007850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.007822 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.009354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.009329 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.065101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.065046 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9v9v\" (UniqueName: \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\") pod \"maas-api-revoke-sub-7eed18ff-chs94\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.112680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.112660 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:01.484510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.484480 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94"] Oct 30 19:34:01.485418 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:01.485390 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeaa48ce_e3f8_4a99_a2f0_7a25301cf681.slice/crio-edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97 WatchSource:0}: Error finding container edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97: Status 404 returned error can't find the container with id edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97 Oct 30 19:34:01.583506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:01.583476 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" event={"ID":"beaa48ce-e3f8-4a99-a2f0-7a25301cf681","Type":"ContainerStarted","Data":"edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97"} Oct 30 19:34:02.592078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:02.591984 4789 generic.go:356] "Generic (PLEG): container finished" podID="beaa48ce-e3f8-4a99-a2f0-7a25301cf681" containerID="09c880094cc7484dc8784d3dde8429c0ab05d5e3f243cfe575d46e9d7538fc42" exitCode=0 Oct 30 19:34:02.592487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:02.592072 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" event={"ID":"beaa48ce-e3f8-4a99-a2f0-7a25301cf681","Type":"ContainerDied","Data":"09c880094cc7484dc8784d3dde8429c0ab05d5e3f243cfe575d46e9d7538fc42"} Oct 30 19:34:03.748545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.748520 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:03.835364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.835335 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\" (UniqueName: \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\") pod \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " Oct 30 19:34:03.835541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.835374 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\") pod \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " Oct 30 19:34:03.835587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.835563 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\") pod \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\" (UID: \"beaa48ce-e3f8-4a99-a2f0-7a25301cf681\") " Oct 30 19:34:03.835897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.835872 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca" pod "beaa48ce-e3f8-4a99-a2f0-7a25301cf681" (UID: "beaa48ce-e3f8-4a99-a2f0-7a25301cf681"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:03.837427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.837402 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca" pod "beaa48ce-e3f8-4a99-a2f0-7a25301cf681" (UID: "beaa48ce-e3f8-4a99-a2f0-7a25301cf681"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:03.837528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.837462 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v" pod "beaa48ce-e3f8-4a99-a2f0-7a25301cf681" (UID: "beaa48ce-e3f8-4a99-a2f0-7a25301cf681"). InnerVolumeSpecName "kube-api-access-h9v9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:03.936928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.936897 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:03.936928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.936926 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9v9v\" (UniqueName: \"kubernetes.io/projected/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-kube-api-access-h9v9v\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:03.937120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:03.936936 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beaa48ce-e3f8-4a99-a2f0-7a25301cf681-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:04.608120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:04.608082 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" event={"ID":"beaa48ce-e3f8-4a99-a2f0-7a25301cf681","Type":"ContainerDied","Data":"edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97"} Oct 30 19:34:04.608120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:04.608120 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94" Oct 30 19:34:04.608365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:04.608142 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edfbf22790620fa784fe48b738352884fbe18c1272bdafee13308eae7e298b97" Oct 30 19:34:06.289540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:06.289511 4789 scope.go:122] "RemoveContainer" containerID="d505911d074246b7fba18992005e30d98c71820859f9d2367ae4292bd9dbb6b0" Oct 30 19:34:06.298751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:06.298733 4789 scope.go:122] "RemoveContainer" containerID="c1bc95c5a019432e6204cf6f4b65c8231067831a552af4778089dddfc1f0aca6" Oct 30 19:34:11.599003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.598966 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm"] Oct 30 19:34:11.599743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.599720 4789 cpu_manager.go:414] "containerMap: removing container" podUID="beaa48ce-e3f8-4a99-a2f0-7a25301cf681" containerName="revoke-keys" Oct 30 19:34:11.599791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.599745 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beaa48ce-e3f8-4a99-a2f0-7a25301cf681" containerName="revoke-keys" Oct 30 19:34:11.599929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.599918 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="beaa48ce-e3f8-4a99-a2f0-7a25301cf681" containerName="revoke-keys" Oct 30 19:34:11.605442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.605418 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.658980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.658943 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm"] Oct 30 19:34:11.701392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.701365 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:11.701572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.701502 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:34:11.701648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.701614 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:11.708701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.708683 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:11.733609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.733574 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.733785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.733641 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.733941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.733915 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2rf\" (UniqueName: \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.834816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.834773 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2rf\" (UniqueName: \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.835035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.834900 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.835035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.834929 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.835676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.835647 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.837527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.837503 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.893810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.893733 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2rf\" (UniqueName: \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\") pod \"maas-api-revoke-sub-d286ebee-4whnm\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:11.920145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:11.920115 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:12.301192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:12.301160 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm"] Oct 30 19:34:12.304298 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:12.304268 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc14a5a6_2507_4eeb_9459_47af1d7abc0c.slice/crio-5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2 WatchSource:0}: Error finding container 5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2: Status 404 returned error can't find the container with id 5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2 Oct 30 19:34:12.660254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:12.660190 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" event={"ID":"bc14a5a6-2507-4eeb-9459-47af1d7abc0c","Type":"ContainerStarted","Data":"e84e72f705d271c856cb68251d559b427c3f3ce5e79f54d1e5a7e934e828afae"} Oct 30 19:34:12.660643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:12.660276 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" event={"ID":"bc14a5a6-2507-4eeb-9459-47af1d7abc0c","Type":"ContainerStarted","Data":"5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2"} Oct 30 19:34:12.769995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:12.769948 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" podStartSLOduration=1.769927641 podStartE2EDuration="1.769927641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:12.769338258 +0000 UTC m=+2889.271248744" watchObservedRunningTime="2026-10-30 19:34:12.769927641 +0000 UTC m=+2889.271838150" Oct 30 19:34:13.670145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:13.670068 4789 generic.go:356] "Generic (PLEG): container finished" podID="bc14a5a6-2507-4eeb-9459-47af1d7abc0c" containerID="e84e72f705d271c856cb68251d559b427c3f3ce5e79f54d1e5a7e934e828afae" exitCode=0 Oct 30 19:34:13.670536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:13.670145 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" event={"ID":"bc14a5a6-2507-4eeb-9459-47af1d7abc0c","Type":"ContainerDied","Data":"e84e72f705d271c856cb68251d559b427c3f3ce5e79f54d1e5a7e934e828afae"} Oct 30 19:34:14.842219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.842181 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:14.968361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.968328 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\" (UniqueName: \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\") pod \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " Oct 30 19:34:14.968737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.968713 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\") pod \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " Oct 30 19:34:14.968919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.968900 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\") pod \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\" (UID: \"bc14a5a6-2507-4eeb-9459-47af1d7abc0c\") " Oct 30 19:34:14.969298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.969270 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca" pod "bc14a5a6-2507-4eeb-9459-47af1d7abc0c" (UID: "bc14a5a6-2507-4eeb-9459-47af1d7abc0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:14.969658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.969636 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:14.971715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.971687 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca" pod "bc14a5a6-2507-4eeb-9459-47af1d7abc0c" (UID: "bc14a5a6-2507-4eeb-9459-47af1d7abc0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:14.976636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:14.976609 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf" pod "bc14a5a6-2507-4eeb-9459-47af1d7abc0c" (UID: "bc14a5a6-2507-4eeb-9459-47af1d7abc0c"). InnerVolumeSpecName "kube-api-access-kw2rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:15.070581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:15.070558 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2rf\" (UniqueName: \"kubernetes.io/projected/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-kube-api-access-kw2rf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:15.070696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:15.070584 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc14a5a6-2507-4eeb-9459-47af1d7abc0c-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:15.687398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:15.687373 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" Oct 30 19:34:15.687545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:15.687398 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm" event={"ID":"bc14a5a6-2507-4eeb-9459-47af1d7abc0c","Type":"ContainerDied","Data":"5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2"} Oct 30 19:34:15.687545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:15.687448 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ad9bdd1b65f14761be857fad4d0209d76fccbdb582670005f3978934296ebc2" Oct 30 19:34:17.617719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.617684 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2"] Oct 30 19:34:17.618297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.618280 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bc14a5a6-2507-4eeb-9459-47af1d7abc0c" containerName="revoke-keys" Oct 30 19:34:17.618361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.618299 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc14a5a6-2507-4eeb-9459-47af1d7abc0c" containerName="revoke-keys" Oct 30 19:34:17.618428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.618419 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc14a5a6-2507-4eeb-9459-47af1d7abc0c" containerName="revoke-keys" Oct 30 19:34:17.621953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.621936 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.718190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.718158 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:17.721413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.721389 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:17.721535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.721495 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:34:17.723175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.723148 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:17.723329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.723180 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2"] Oct 30 19:34:17.798985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.798955 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.799075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.799020 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.799146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.799123 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjf5z\" (UniqueName: \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.900079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.900023 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.900079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.900068 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.900236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.900105 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjf5z\" (UniqueName: \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.900809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.900792 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.902363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.902340 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:17.957678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:17.957655 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjf5z\" (UniqueName: \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\") pod \"maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:18.196248 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:18.196200 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:34:18.237638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:18.237611 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:18.610322 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:18.610289 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2"] Oct 30 19:34:18.611863 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:18.611833 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c44c980_1aa1_4d90_a215_e7e511c1954b.slice/crio-371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241 WatchSource:0}: Error finding container 371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241: Status 404 returned error can't find the container with id 371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241 Oct 30 19:34:18.729978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:18.729942 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" event={"ID":"6c44c980-1aa1-4d90-a215-e7e511c1954b","Type":"ContainerStarted","Data":"371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241"} Oct 30 19:34:19.737077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:19.737004 4789 generic.go:356] "Generic (PLEG): container finished" podID="6c44c980-1aa1-4d90-a215-e7e511c1954b" containerID="3f4efcf56bf935413c6d04652ca3eb2436db3d189061f0f57b2e45f8337750c8" exitCode=0 Oct 30 19:34:19.737477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:19.737081 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" event={"ID":"6c44c980-1aa1-4d90-a215-e7e511c1954b","Type":"ContainerDied","Data":"3f4efcf56bf935413c6d04652ca3eb2436db3d189061f0f57b2e45f8337750c8"} Oct 30 19:34:20.880872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:20.880849 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:21.030836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.030705 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\") pod \"6c44c980-1aa1-4d90-a215-e7e511c1954b\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " Oct 30 19:34:21.030973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.030869 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\" (UniqueName: \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\") pod \"6c44c980-1aa1-4d90-a215-e7e511c1954b\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " Oct 30 19:34:21.030973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.030938 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\") pod \"6c44c980-1aa1-4d90-a215-e7e511c1954b\" (UID: \"6c44c980-1aa1-4d90-a215-e7e511c1954b\") " Oct 30 19:34:21.031055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.030983 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca" pod "6c44c980-1aa1-4d90-a215-e7e511c1954b" (UID: "6c44c980-1aa1-4d90-a215-e7e511c1954b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:21.031271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.031250 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:21.032917 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.032895 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z" pod "6c44c980-1aa1-4d90-a215-e7e511c1954b" (UID: "6c44c980-1aa1-4d90-a215-e7e511c1954b"). InnerVolumeSpecName "kube-api-access-kjf5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:21.032998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.032912 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca" pod "6c44c980-1aa1-4d90-a215-e7e511c1954b" (UID: "6c44c980-1aa1-4d90-a215-e7e511c1954b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:21.132883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.132852 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjf5z\" (UniqueName: \"kubernetes.io/projected/6c44c980-1aa1-4d90-a215-e7e511c1954b-kube-api-access-kjf5z\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:21.132883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.132884 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c44c980-1aa1-4d90-a215-e7e511c1954b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:21.750517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.750487 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" Oct 30 19:34:21.750517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.750498 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2" event={"ID":"6c44c980-1aa1-4d90-a215-e7e511c1954b","Type":"ContainerDied","Data":"371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241"} Oct 30 19:34:21.750693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.750540 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371bfcbb031436c6ccf06b7e2e7de0a8f86f1ad774f21d04cbedecfae3fcd241" Oct 30 19:34:21.854433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.854404 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4"] Oct 30 19:34:21.854893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.854878 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6c44c980-1aa1-4d90-a215-e7e511c1954b" containerName="revoke-keys" Oct 30 19:34:21.854893 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.854892 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c44c980-1aa1-4d90-a215-e7e511c1954b" containerName="revoke-keys" Oct 30 19:34:21.855038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.854989 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c44c980-1aa1-4d90-a215-e7e511c1954b" containerName="revoke-keys" Oct 30 19:34:21.858509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.858492 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:21.901725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.901698 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4"] Oct 30 19:34:21.907628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.907603 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:21.907775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.907636 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:21.907775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.907704 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:34:21.907775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:21.907726 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:22.040976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.040904 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.040976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.040963 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.041179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.041156 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6zv\" (UniqueName: \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.141775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.141749 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.141918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.141825 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6zv\" (UniqueName: \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.141918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.141881 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.142460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.142430 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.144162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.144135 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.199874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.199841 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6zv\" (UniqueName: \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\") pod \"maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.472098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.472075 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:22.849620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:22.849586 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4"] Oct 30 19:34:22.851757 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:22.851732 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4c6ddb_46b4_49bf_859d_e2b8529dd6e0.slice/crio-dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212 WatchSource:0}: Error finding container dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212: Status 404 returned error can't find the container with id dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212 Oct 30 19:34:23.100611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.100548 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49"] Oct 30 19:34:23.106221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.106180 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.161970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.161946 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49"] Oct 30 19:34:23.251379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.251353 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.251519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.251403 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.251593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.251522 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2nq9\" (UniqueName: \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.352671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.352602 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.352671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.352647 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.352852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.352761 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2nq9\" (UniqueName: \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.353276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.353257 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.354975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.354953 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.410961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.410930 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2nq9\" (UniqueName: \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\") pod \"maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.420766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.420744 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:23.765422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.765390 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" event={"ID":"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0","Type":"ContainerStarted","Data":"76376c897275a3d3c49962b4c1f1b1b0c5df85823ed8e3ec36c06986df307cfc"} Oct 30 19:34:23.765594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.765442 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" event={"ID":"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0","Type":"ContainerStarted","Data":"dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212"} Oct 30 19:34:23.797596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.797572 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49"] Oct 30 19:34:23.835021 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:23.834995 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7d0528_242e_4b4a_857d_0e19b177bff1.slice/crio-e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f WatchSource:0}: Error finding container e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f: Status 404 returned error can't find the container with id e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f Oct 30 19:34:23.879066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:23.879027 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" podStartSLOduration=2.879013745 podStartE2EDuration="2.879013745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:23.875194957 +0000 UTC m=+2900.377105442" watchObservedRunningTime="2026-10-30 19:34:23.879013745 +0000 UTC m=+2900.380924230" Oct 30 19:34:24.774046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:24.774017 4789 generic.go:356] "Generic (PLEG): container finished" podID="1b7d0528-242e-4b4a-857d-0e19b177bff1" containerID="56c883f76d79a4aa7e9ad254eb8a56203a035db3522e8095257850dcd3628ba6" exitCode=0 Oct 30 19:34:24.774462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:24.774104 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" event={"ID":"1b7d0528-242e-4b4a-857d-0e19b177bff1","Type":"ContainerDied","Data":"56c883f76d79a4aa7e9ad254eb8a56203a035db3522e8095257850dcd3628ba6"} Oct 30 19:34:24.774462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:24.774158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" event={"ID":"1b7d0528-242e-4b4a-857d-0e19b177bff1","Type":"ContainerStarted","Data":"e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f"} Oct 30 19:34:24.775862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:24.775838 4789 generic.go:356] "Generic (PLEG): container finished" podID="4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" containerID="76376c897275a3d3c49962b4c1f1b1b0c5df85823ed8e3ec36c06986df307cfc" exitCode=0 Oct 30 19:34:24.775980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:24.775919 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" event={"ID":"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0","Type":"ContainerDied","Data":"76376c897275a3d3c49962b4c1f1b1b0c5df85823ed8e3ec36c06986df307cfc"} Oct 30 19:34:25.959584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.959553 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:25.978636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.978611 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\") pod \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " Oct 30 19:34:25.978755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.978710 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\" (UniqueName: \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\") pod \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " Oct 30 19:34:25.978802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.978783 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\") pod \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\" (UID: \"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0\") " Oct 30 19:34:25.984462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.979511 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca" pod "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" (UID: "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:25.984462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.980930 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:25.984462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.981550 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca" pod "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" (UID: "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:25.984462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.982102 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv" pod "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" (UID: "4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0"). InnerVolumeSpecName "kube-api-access-8v6zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:25.991459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:25.991433 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:26.081631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.081554 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\") pod \"1b7d0528-242e-4b4a-857d-0e19b177bff1\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " Oct 30 19:34:26.081631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.081622 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\" (UniqueName: \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\") pod \"1b7d0528-242e-4b4a-857d-0e19b177bff1\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " Oct 30 19:34:26.081872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.081811 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\") pod \"1b7d0528-242e-4b4a-857d-0e19b177bff1\" (UID: \"1b7d0528-242e-4b4a-857d-0e19b177bff1\") " Oct 30 19:34:26.081973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.081949 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca" pod "1b7d0528-242e-4b4a-857d-0e19b177bff1" (UID: "1b7d0528-242e-4b4a-857d-0e19b177bff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:26.082200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.082183 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.082275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.082222 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.082275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.082239 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v6zv\" (UniqueName: \"kubernetes.io/projected/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0-kube-api-access-8v6zv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.084132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.084107 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9" pod "1b7d0528-242e-4b4a-857d-0e19b177bff1" (UID: "1b7d0528-242e-4b4a-857d-0e19b177bff1"). InnerVolumeSpecName "kube-api-access-v2nq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:26.084392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.084372 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca" pod "1b7d0528-242e-4b4a-857d-0e19b177bff1" (UID: "1b7d0528-242e-4b4a-857d-0e19b177bff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:26.183240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.183174 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2nq9\" (UniqueName: \"kubernetes.io/projected/1b7d0528-242e-4b4a-857d-0e19b177bff1-kube-api-access-v2nq9\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.183240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.183232 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7d0528-242e-4b4a-857d-0e19b177bff1-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.790895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.790868 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" Oct 30 19:34:26.791075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.790898 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49" event={"ID":"1b7d0528-242e-4b4a-857d-0e19b177bff1","Type":"ContainerDied","Data":"e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f"} Oct 30 19:34:26.791075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.790943 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2263b3896e3fd6b90e5eed0c10496b38bc428aed2d3f106fc95f58cec4e598f" Oct 30 19:34:26.792606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.792584 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" Oct 30 19:34:26.792720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.792601 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4" event={"ID":"4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0","Type":"ContainerDied","Data":"dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212"} Oct 30 19:34:26.792720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:26.792642 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac7d5dd7dae95845c26f92ebd20e25b50f7ac2fba21c42aaae6fafd74c1f212" Oct 30 19:34:29.027225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:29.027170 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:34:29.125949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:29.125918 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:34:33.259856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:33.259819 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:34:33.369644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:33.369611 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:34:34.028441 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.028403 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" podUID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" containerName="payload-processing" containerID="cri-o://5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0" gracePeriod=25 Oct 30 19:34:34.126304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.126250 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" podUID="93142e47-3f86-4a30-b03a-c5e205e37f46" containerName="payload-pre-processing" containerID="cri-o://f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd" gracePeriod=25 Oct 30 19:34:34.329114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.329084 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:34:34.373364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.373328 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\" (UniqueName: \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\") pod \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " Oct 30 19:34:34.373558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.373436 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\") pod \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\" (UID: \"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee\") " Oct 30 19:34:34.373898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.373862 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume" pod "d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" (UID: "d14ab7b6-3a4d-4ac8-84f4-078a62a540ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:34.376465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.376443 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt" pod "d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" (UID: "d14ab7b6-3a4d-4ac8-84f4-078a62a540ee"). InnerVolumeSpecName "kube-api-access-2zdpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:34.402479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.402457 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:34:34.475088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.475050 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\" (UniqueName: \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\") pod \"93142e47-3f86-4a30-b03a-c5e205e37f46\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " Oct 30 19:34:34.475277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.475155 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\") pod \"93142e47-3f86-4a30-b03a-c5e205e37f46\" (UID: \"93142e47-3f86-4a30-b03a-c5e205e37f46\") " Oct 30 19:34:34.475633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.475604 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zdpt\" (UniqueName: \"kubernetes.io/projected/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-kube-api-access-2zdpt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.475633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.475629 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.475828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.475637 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume" pod "93142e47-3f86-4a30-b03a-c5e205e37f46" (UID: "93142e47-3f86-4a30-b03a-c5e205e37f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:34.477937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.477915 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw" pod "93142e47-3f86-4a30-b03a-c5e205e37f46" (UID: "93142e47-3f86-4a30-b03a-c5e205e37f46"). InnerVolumeSpecName "kube-api-access-49dkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:34.577054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.576977 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dkw\" (UniqueName: \"kubernetes.io/projected/93142e47-3f86-4a30-b03a-c5e205e37f46-kube-api-access-49dkw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.577054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.577014 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93142e47-3f86-4a30-b03a-c5e205e37f46-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.864014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.863926 4789 generic.go:356] "Generic (PLEG): container finished" podID="93142e47-3f86-4a30-b03a-c5e205e37f46" containerID="f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd" exitCode=0 Oct 30 19:34:34.864190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.864009 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" Oct 30 19:34:34.864190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.864033 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" event={"ID":"93142e47-3f86-4a30-b03a-c5e205e37f46","Type":"ContainerDied","Data":"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd"} Oct 30 19:34:34.864190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.864092 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f" event={"ID":"93142e47-3f86-4a30-b03a-c5e205e37f46","Type":"ContainerDied","Data":"e7734ed96c4738adbd6cd3b3fe1415c538394c82e2531d32f50263096f9df385"} Oct 30 19:34:34.864190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.864124 4789 scope.go:122] "RemoveContainer" containerID="f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd" Oct 30 19:34:34.865613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.865592 4789 generic.go:356] "Generic (PLEG): container finished" podID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" containerID="5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0" exitCode=0 Oct 30 19:34:34.865737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.865623 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" event={"ID":"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee","Type":"ContainerDied","Data":"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0"} Oct 30 19:34:34.865737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.865649 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" Oct 30 19:34:34.865737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.865653 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5" event={"ID":"d14ab7b6-3a4d-4ac8-84f4-078a62a540ee","Type":"ContainerDied","Data":"1ebf263e10899ee1f0b49a7c419a290328e71f0e769b8955bbfc56687a4aaad0"} Oct 30 19:34:34.874466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.874443 4789 scope.go:122] "RemoveContainer" containerID="f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd" Oct 30 19:34:34.874808 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:34.874785 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd\": container with ID starting with f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd not found: ID does not exist" containerID="f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd" Oct 30 19:34:34.874927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.874814 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd"} err="failed to get container status \"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd\": rpc error: code = NotFound desc = could not find container \"f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd\": container with ID starting with f5adfc78d15adf153d8127cfc137c037699049bacd97f2ff239fba091d9ba1dd not found: ID does not exist" Oct 30 19:34:34.874927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.874832 4789 scope.go:122] "RemoveContainer" containerID="5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0" Oct 30 19:34:34.885035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.885012 4789 scope.go:122] "RemoveContainer" containerID="5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0" Oct 30 19:34:34.885393 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:34.885373 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0\": container with ID starting with 5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0 not found: ID does not exist" containerID="5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0" Oct 30 19:34:34.885477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:34.885401 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0"} err="failed to get container status \"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0\": rpc error: code = NotFound desc = could not find container \"5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0\": container with ID starting with 5034c44f7e64d598a56e3ac24da85d6610b202bbd1037c3abeca6bed209eead0 not found: ID does not exist" Oct 30 19:34:35.019219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.019165 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:34:35.065447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.065407 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e73f1a-58fb4f999b-768w5"] Oct 30 19:34:35.128464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.128374 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:34:35.134474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.134443 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e73f1a-78cd6984c5-jlk6f"] Oct 30 19:34:35.669078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.669046 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93142e47-3f86-4a30-b03a-c5e205e37f46" path="/var/lib/kubelet/pods/93142e47-3f86-4a30-b03a-c5e205e37f46/volumes" Oct 30 19:34:35.669501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:35.669462 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" path="/var/lib/kubelet/pods/d14ab7b6-3a4d-4ac8-84f4-078a62a540ee/volumes" Oct 30 19:34:38.092102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.091967 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h"] Oct 30 19:34:38.092608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092589 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1b7d0528-242e-4b4a-857d-0e19b177bff1" containerName="revoke-keys" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092608 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7d0528-242e-4b4a-857d-0e19b177bff1" containerName="revoke-keys" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092626 4789 cpu_manager.go:414] "containerMap: removing container" podUID="93142e47-3f86-4a30-b03a-c5e205e37f46" containerName="payload-pre-processing" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092632 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93142e47-3f86-4a30-b03a-c5e205e37f46" containerName="payload-pre-processing" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092651 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" containerName="payload-processing" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092656 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" containerName="payload-processing" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092669 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" containerName="revoke-keys" Oct 30 19:34:38.092670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092674 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" containerName="revoke-keys" Oct 30 19:34:38.093055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092752 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" containerName="revoke-keys" Oct 30 19:34:38.093055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092765 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7d0528-242e-4b4a-857d-0e19b177bff1" containerName="revoke-keys" Oct 30 19:34:38.093055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092773 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="93142e47-3f86-4a30-b03a-c5e205e37f46" containerName="payload-pre-processing" Oct 30 19:34:38.093055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.092780 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14ab7b6-3a4d-4ac8-84f4-078a62a540ee" containerName="payload-processing" Oct 30 19:34:38.097323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.097301 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.194169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.194117 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:38.194363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.194193 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:38.194363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.194240 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:38.194363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.194351 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:34:38.198952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.198927 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h"] Oct 30 19:34:38.220920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.220888 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.221048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.220976 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.221048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.221030 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcxv\" (UniqueName: \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.261034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.261001 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" podUID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" containerName="payload-processing" containerID="cri-o://6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981" gracePeriod=25 Oct 30 19:34:38.321913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.321878 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.322090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.321928 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.322090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.321959 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcxv\" (UniqueName: \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.322723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.322693 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.324531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.324507 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.370902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.370829 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" podUID="a9cca9a3-f612-4f2a-87c7-688892c618d7" containerName="payload-pre-processing" containerID="cri-o://b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034" gracePeriod=25 Oct 30 19:34:38.380163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.380141 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcxv\" (UniqueName: \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\") pod \"maas-api-revoke-sub-795aed21-9sw5h\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.413285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.413251 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:38.540838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.539805 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:34:38.629490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.629398 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\" (UniqueName: \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\") pod \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " Oct 30 19:34:38.629668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.629525 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\") pod \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\" (UID: \"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc\") " Oct 30 19:34:38.630248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.630201 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume" pod "c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" (UID: "c6231fa9-65ba-4d93-8aa9-4161e52d1ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:38.632629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.632606 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp" pod "c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" (UID: "c6231fa9-65ba-4d93-8aa9-4161e52d1ffc"). InnerVolumeSpecName "kube-api-access-875rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:38.656770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.656738 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:34:38.731265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.731226 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\" (UniqueName: \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\") pod \"a9cca9a3-f612-4f2a-87c7-688892c618d7\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " Oct 30 19:34:38.731481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.731427 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\") pod \"a9cca9a3-f612-4f2a-87c7-688892c618d7\" (UID: \"a9cca9a3-f612-4f2a-87c7-688892c618d7\") " Oct 30 19:34:38.731829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.731803 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume" pod "a9cca9a3-f612-4f2a-87c7-688892c618d7" (UID: "a9cca9a3-f612-4f2a-87c7-688892c618d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:38.731829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.731821 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875rp\" (UniqueName: \"kubernetes.io/projected/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-kube-api-access-875rp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:38.731954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.731839 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:38.733446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.733422 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5" pod "a9cca9a3-f612-4f2a-87c7-688892c618d7" (UID: "a9cca9a3-f612-4f2a-87c7-688892c618d7"). InnerVolumeSpecName "kube-api-access-x4wx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:38.801842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.801808 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h"] Oct 30 19:34:38.802456 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:38.802427 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccae1a1c_ec7b_4196_8793_5e47268af4c6.slice/crio-f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6 WatchSource:0}: Error finding container f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6: Status 404 returned error can't find the container with id f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6 Oct 30 19:34:38.832996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.832965 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4wx5\" (UniqueName: \"kubernetes.io/projected/a9cca9a3-f612-4f2a-87c7-688892c618d7-kube-api-access-x4wx5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:38.833125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.833000 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cca9a3-f612-4f2a-87c7-688892c618d7-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:38.902014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.901917 4789 generic.go:356] "Generic (PLEG): container finished" podID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" containerID="6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981" exitCode=0 Oct 30 19:34:38.902014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.901973 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" Oct 30 19:34:38.902014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.902000 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" event={"ID":"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc","Type":"ContainerDied","Data":"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981"} Oct 30 19:34:38.902309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.902080 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff" event={"ID":"c6231fa9-65ba-4d93-8aa9-4161e52d1ffc","Type":"ContainerDied","Data":"a0890122eac48d565a75b8899adcca3cdf4d306166dac3ac0d88bdb9adcb6aea"} Oct 30 19:34:38.902309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.902108 4789 scope.go:122] "RemoveContainer" containerID="6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981" Oct 30 19:34:38.903961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.903935 4789 generic.go:356] "Generic (PLEG): container finished" podID="a9cca9a3-f612-4f2a-87c7-688892c618d7" containerID="b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034" exitCode=0 Oct 30 19:34:38.904056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.904001 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" Oct 30 19:34:38.904056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.904022 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" event={"ID":"a9cca9a3-f612-4f2a-87c7-688892c618d7","Type":"ContainerDied","Data":"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034"} Oct 30 19:34:38.904176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.904072 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98" event={"ID":"a9cca9a3-f612-4f2a-87c7-688892c618d7","Type":"ContainerDied","Data":"fad71ccac3f681a5ecf3204984b4d9bfb91b675c5f3515c56862a406cdf79de5"} Oct 30 19:34:38.906058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.906030 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" event={"ID":"ccae1a1c-ec7b-4196-8793-5e47268af4c6","Type":"ContainerStarted","Data":"f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6"} Oct 30 19:34:38.916324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.916300 4789 scope.go:122] "RemoveContainer" containerID="6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981" Oct 30 19:34:38.916680 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:38.916656 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981\": container with ID starting with 6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981 not found: ID does not exist" containerID="6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981" Oct 30 19:34:38.916766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.916691 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981"} err="failed to get container status \"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981\": rpc error: code = NotFound desc = could not find container \"6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981\": container with ID starting with 6444a272e9d42c3469ecbcae354dfb89f3d6996b68a57fbc81f9631785d91981 not found: ID does not exist" Oct 30 19:34:38.916766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.916716 4789 scope.go:122] "RemoveContainer" containerID="b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034" Oct 30 19:34:38.928021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.927996 4789 scope.go:122] "RemoveContainer" containerID="b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034" Oct 30 19:34:38.928379 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:38.928354 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034\": container with ID starting with b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034 not found: ID does not exist" containerID="b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034" Oct 30 19:34:38.928445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:38.928390 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034"} err="failed to get container status \"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034\": rpc error: code = NotFound desc = could not find container \"b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034\": container with ID starting with b9f4d9fab74793b4bb53e52c0ca70dfadb81ad487b1861a12453822832401034 not found: ID does not exist" Oct 30 19:34:39.057843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.057808 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:34:39.062263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.062234 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-69d429-56f84d5547-gph98"] Oct 30 19:34:39.165331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.165254 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:34:39.170115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.170088 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-69d429-7b75595f4-77kff"] Oct 30 19:34:39.668062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.668029 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cca9a3-f612-4f2a-87c7-688892c618d7" path="/var/lib/kubelet/pods/a9cca9a3-f612-4f2a-87c7-688892c618d7/volumes" Oct 30 19:34:39.668467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.668452 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" path="/var/lib/kubelet/pods/c6231fa9-65ba-4d93-8aa9-4161e52d1ffc/volumes" Oct 30 19:34:39.920202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.920120 4789 generic.go:356] "Generic (PLEG): container finished" podID="ccae1a1c-ec7b-4196-8793-5e47268af4c6" containerID="a1670c13ea2b69e7ea14f2e5cdaa31a317cba5620bf7e47b41e7122339ef8c26" exitCode=0 Oct 30 19:34:39.920369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:39.920201 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" event={"ID":"ccae1a1c-ec7b-4196-8793-5e47268af4c6","Type":"ContainerDied","Data":"a1670c13ea2b69e7ea14f2e5cdaa31a317cba5620bf7e47b41e7122339ef8c26"} Oct 30 19:34:41.084358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.084329 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:41.159934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.159906 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\") pod \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " Oct 30 19:34:41.160088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.159947 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\") pod \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " Oct 30 19:34:41.160088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.159989 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\" (UniqueName: \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\") pod \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\" (UID: \"ccae1a1c-ec7b-4196-8793-5e47268af4c6\") " Oct 30 19:34:41.160454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.160422 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca" pod "ccae1a1c-ec7b-4196-8793-5e47268af4c6" (UID: "ccae1a1c-ec7b-4196-8793-5e47268af4c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:41.162015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.161990 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca" pod "ccae1a1c-ec7b-4196-8793-5e47268af4c6" (UID: "ccae1a1c-ec7b-4196-8793-5e47268af4c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:41.162077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.162053 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv" pod "ccae1a1c-ec7b-4196-8793-5e47268af4c6" (UID: "ccae1a1c-ec7b-4196-8793-5e47268af4c6"). InnerVolumeSpecName "kube-api-access-7gcxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:41.261871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.261841 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:41.261871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.261867 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcxv\" (UniqueName: \"kubernetes.io/projected/ccae1a1c-ec7b-4196-8793-5e47268af4c6-kube-api-access-7gcxv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:41.261871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.261878 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccae1a1c-ec7b-4196-8793-5e47268af4c6-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:41.934276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.934246 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" Oct 30 19:34:41.934276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.934262 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h" event={"ID":"ccae1a1c-ec7b-4196-8793-5e47268af4c6","Type":"ContainerDied","Data":"f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6"} Oct 30 19:34:41.934477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:41.934310 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3dd286407ff8cd9610af353b55f00bf390dcd3d2a10bf5296848513c60d53d6" Oct 30 19:34:47.113394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:47.113364 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4"] Oct 30 19:34:47.122463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:47.122441 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-69d429-l4cl4"] Oct 30 19:34:47.668358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:47.668324 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0" path="/var/lib/kubelet/pods/4b4c6ddb-46b4-49bf-859d-e2b8529dd6e0/volumes" Oct 30 19:34:48.196775 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:34:48.196729 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:34:52.160063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160025 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:34:52.160757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160736 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ccae1a1c-ec7b-4196-8793-5e47268af4c6" containerName="revoke-keys" Oct 30 19:34:52.160804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160760 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccae1a1c-ec7b-4196-8793-5e47268af4c6" containerName="revoke-keys" Oct 30 19:34:52.160804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160798 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a9cca9a3-f612-4f2a-87c7-688892c618d7" containerName="payload-pre-processing" Oct 30 19:34:52.160875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160806 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cca9a3-f612-4f2a-87c7-688892c618d7" containerName="payload-pre-processing" Oct 30 19:34:52.160875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160821 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" containerName="payload-processing" Oct 30 19:34:52.160875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160828 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" containerName="payload-processing" Oct 30 19:34:52.161015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160952 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cca9a3-f612-4f2a-87c7-688892c618d7" containerName="payload-pre-processing" Oct 30 19:34:52.161015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160963 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6231fa9-65ba-4d93-8aa9-4161e52d1ffc" containerName="payload-processing" Oct 30 19:34:52.161015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.160983 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccae1a1c-ec7b-4196-8793-5e47268af4c6" containerName="revoke-keys" Oct 30 19:34:52.166053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.166024 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282288 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282360 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282389 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf5w\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282443 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282471 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282501 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282550 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282600 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.282801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.282626 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.283522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.283497 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:34:52.383928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.383900 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.383928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.383934 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.383954 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.383982 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384009 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384025 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384065 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384127 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384173 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf5w\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384510 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384523 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384609 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384736 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.384977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.384955 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.386353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.386327 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.386491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.386471 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.443115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.443092 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.443427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.443404 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf5w\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.480385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.480363 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:52.906566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.906536 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:34:52.908506 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:34:52.908474 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a18a7cf_efc4_451c_9d94_7d6213eaec43.slice/crio-3f204607a6748ac7bc55b3b12710b1eaba2e88f266d2a73a6bedfd0020dedb30 WatchSource:0}: Error finding container 3f204607a6748ac7bc55b3b12710b1eaba2e88f266d2a73a6bedfd0020dedb30: Status 404 returned error can't find the container with id 3f204607a6748ac7bc55b3b12710b1eaba2e88f266d2a73a6bedfd0020dedb30 Oct 30 19:34:52.912041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.912003 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:52.912139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.912088 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:52.912139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:52.912131 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:34:53.012743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.012710 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" event={"ID":"6a18a7cf-efc4-451c-9d94-7d6213eaec43","Type":"ContainerStarted","Data":"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26"} Oct 30 19:34:53.013192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.012772 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" event={"ID":"6a18a7cf-efc4-451c-9d94-7d6213eaec43","Type":"ContainerStarted","Data":"3f204607a6748ac7bc55b3b12710b1eaba2e88f266d2a73a6bedfd0020dedb30"} Oct 30 19:34:53.135204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.135135 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podStartSLOduration=1.13511721 podStartE2EDuration="1.13511721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:53.134181764 +0000 UTC m=+2929.636092255" watchObservedRunningTime="2026-10-30 19:34:53.13511721 +0000 UTC m=+2929.637027697" Oct 30 19:34:53.480545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.480513 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:34:53.482270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.482246 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:53.482412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:53.482290 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:54.481655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:54.481609 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:54.482104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:54.481682 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:54.657153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:54.657113 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:34:54.657455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:54.657426 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" containerID="cri-o://a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c" gracePeriod=30 Oct 30 19:34:55.481303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:55.481267 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:55.481498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:55.481333 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:56.481722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:56.481689 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:56.482137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:56.481746 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:57.480878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:57.480840 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:57.481051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:57.480902 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:58.481517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:58.481481 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:58.482009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:58.481536 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:59.481451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.481413 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 19:34:59.481663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.481488 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 19:34:59.933262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.933239 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:34:59.995276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995242 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995316 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995377 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995446 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995490 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995618 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:59.995722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995625 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995734 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995770 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.995901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.995813 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\") pod \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\" (UID: \"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11\") " Oct 30 19:34:59.996422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.996319 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:59.996422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.996341 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:59.996422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.996342 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:59.996422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.996398 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:59.996875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.996852 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:59.997938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.997913 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "kube-api-access-b9jjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:59.998360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.998339 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:59.998608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.998590 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:59.998914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:34:59.998892 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo" pod "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" (UID: "fa5a4780-67c7-4c14-b14a-c9a6acdf9e11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:00.076810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.076462 4789 generic.go:356] "Generic (PLEG): container finished" podID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerID="a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c" exitCode=0 Oct 30 19:35:00.076810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.076524 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerDied","Data":"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c"} Oct 30 19:35:00.076810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.076579 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" event={"ID":"fa5a4780-67c7-4c14-b14a-c9a6acdf9e11","Type":"ContainerDied","Data":"c9b4140b7e3b389e0662aed52054e639f87f5b03dc590ff5c33447d0c41cb1b0"} Oct 30 19:35:00.076810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.076529 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd" Oct 30 19:35:00.076810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.076613 4789 scope.go:122] "RemoveContainer" containerID="a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c" Oct 30 19:35:00.092301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.092277 4789 scope.go:122] "RemoveContainer" containerID="87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc" Oct 30 19:35:00.098104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098074 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098111 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098128 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098145 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098160 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098174 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098189 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.098613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.098203 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jjn\" (UniqueName: \"kubernetes.io/projected/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11-kube-api-access-b9jjn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:00.114398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.114371 4789 scope.go:122] "RemoveContainer" containerID="a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c" Oct 30 19:35:00.114942 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:00.114912 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c\": container with ID starting with a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c not found: ID does not exist" containerID="a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c" Oct 30 19:35:00.115064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.114953 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c"} err="failed to get container status \"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c\": rpc error: code = NotFound desc = could not find container \"a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c\": container with ID starting with a7d86afbcd5706e0c4a5447fb3d70d2f42eda9e726d1694d092f82cc4d1f271c not found: ID does not exist" Oct 30 19:35:00.115064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.114977 4789 scope.go:122] "RemoveContainer" containerID="87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc" Oct 30 19:35:00.115475 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:00.115439 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc\": container with ID starting with 87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc not found: ID does not exist" containerID="87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc" Oct 30 19:35:00.115586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.115487 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc"} err="failed to get container status \"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc\": rpc error: code = NotFound desc = could not find container \"87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc\": container with ID starting with 87cd03004a31639a544a61c77ad9aa4df0895fab77a29f378d38757da1ede8cc not found: ID does not exist" Oct 30 19:35:00.235820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.235782 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:35:00.241178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.241148 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-69d429-openshift-default-75b74895d5-5v2nd"] Oct 30 19:35:00.484986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.484959 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:35:00.485513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.485455 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:35:00.486237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:00.486193 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:35:01.668549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:01.668519 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" path="/var/lib/kubelet/pods/fa5a4780-67c7-4c14-b14a-c9a6acdf9e11/volumes" Oct 30 19:35:07.442848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:07.442813 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2"] Oct 30 19:35:07.487147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:07.487116 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e73f1a-5d8v2"] Oct 30 19:35:07.668072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:07.668040 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c44c980-1aa1-4d90-a215-e7e511c1954b" path="/var/lib/kubelet/pods/6c44c980-1aa1-4d90-a215-e7e511c1954b/volumes" Oct 30 19:35:10.715597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.715558 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5"] Oct 30 19:35:10.716155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716133 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.716203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716158 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.716203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716182 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.716203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716190 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.716347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716335 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.716392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.716351 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5a4780-67c7-4c14-b14a-c9a6acdf9e11" containerName="istio-proxy" Oct 30 19:35:10.720734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.720719 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.817468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.817436 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:10.817614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.817540 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:10.817614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.817560 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:10.817835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.817809 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:35:10.820493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.820471 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5"] Oct 30 19:35:10.822928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.822906 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.823031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.823014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.823094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.823087 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4rk\" (UniqueName: \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.923774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.923749 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4rk\" (UniqueName: \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.923867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.923799 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.923867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.923843 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.924586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.924568 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.926112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.926094 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:10.981660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:10.981613 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4rk\" (UniqueName: \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\") pod \"maas-api-revoke-sub-71d9477a-68rz5\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:11.015938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.015919 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb"] Oct 30 19:35:11.019916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.019899 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.035760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.035743 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:11.079662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.079630 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb"] Oct 30 19:35:11.125179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.125150 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6cm\" (UniqueName: \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.125324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.125241 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.125368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.125353 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.226325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.226295 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6cm\" (UniqueName: \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.226450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.226345 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.226450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.226389 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.227101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.227082 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.228788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.228766 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.284013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.283960 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6cm\" (UniqueName: \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\") pod \"maas-api-revoke-sub-a2c3b3f9-6lgjb\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.332908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.332888 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:11.411248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.410169 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5"] Oct 30 19:35:11.413358 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:11.413332 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c58ce6b_2074_478d_8400_970134c9c081.slice/crio-24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878 WatchSource:0}: Error finding container 24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878: Status 404 returned error can't find the container with id 24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878 Oct 30 19:35:11.708008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:11.707981 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb"] Oct 30 19:35:11.708900 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:11.708876 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45682fdf_06d1_476e_8e5d_750cf4a32b4e.slice/crio-2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85 WatchSource:0}: Error finding container 2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85: Status 404 returned error can't find the container with id 2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85 Oct 30 19:35:12.162446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.162358 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" event={"ID":"45682fdf-06d1-476e-8e5d-750cf4a32b4e","Type":"ContainerStarted","Data":"26b5797685b4b6c02451c1e59b59e1f465255c821c5b25b352bfca9fa96c1a1d"} Oct 30 19:35:12.162446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.162424 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" event={"ID":"45682fdf-06d1-476e-8e5d-750cf4a32b4e","Type":"ContainerStarted","Data":"2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85"} Oct 30 19:35:12.164082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.164058 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" event={"ID":"8c58ce6b-2074-478d-8400-970134c9c081","Type":"ContainerStarted","Data":"afa1f11ee76fad693bb0d0b94bfc998bb31ec59aa0e72a3764a9e2236dfae07a"} Oct 30 19:35:12.164197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.164097 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" event={"ID":"8c58ce6b-2074-478d-8400-970134c9c081","Type":"ContainerStarted","Data":"24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878"} Oct 30 19:35:12.273753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.273712 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" podStartSLOduration=2.273698608 podStartE2EDuration="2.273698608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.271342424 +0000 UTC m=+2948.773252921" watchObservedRunningTime="2026-10-30 19:35:12.273698608 +0000 UTC m=+2948.775609088" Oct 30 19:35:12.382980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:12.382939 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" podStartSLOduration=2.382926768 podStartE2EDuration="2.382926768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:12.380302266 +0000 UTC m=+2948.882212753" watchObservedRunningTime="2026-10-30 19:35:12.382926768 +0000 UTC m=+2948.884837253" Oct 30 19:35:13.192879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:13.192839 4789 generic.go:356] "Generic (PLEG): container finished" podID="45682fdf-06d1-476e-8e5d-750cf4a32b4e" containerID="26b5797685b4b6c02451c1e59b59e1f465255c821c5b25b352bfca9fa96c1a1d" exitCode=0 Oct 30 19:35:13.193415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:13.192949 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" event={"ID":"45682fdf-06d1-476e-8e5d-750cf4a32b4e","Type":"ContainerDied","Data":"26b5797685b4b6c02451c1e59b59e1f465255c821c5b25b352bfca9fa96c1a1d"} Oct 30 19:35:13.195047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:13.195022 4789 generic.go:356] "Generic (PLEG): container finished" podID="8c58ce6b-2074-478d-8400-970134c9c081" containerID="afa1f11ee76fad693bb0d0b94bfc998bb31ec59aa0e72a3764a9e2236dfae07a" exitCode=0 Oct 30 19:35:13.195325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:13.195114 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" event={"ID":"8c58ce6b-2074-478d-8400-970134c9c081","Type":"ContainerDied","Data":"afa1f11ee76fad693bb0d0b94bfc998bb31ec59aa0e72a3764a9e2236dfae07a"} Oct 30 19:35:14.006070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.006035 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:35:14.106761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.106727 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:35:14.379689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.379662 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:14.429922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.429260 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:14.459223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.459180 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\") pod \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " Oct 30 19:35:14.459401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.459306 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\") pod \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " Oct 30 19:35:14.459401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.459338 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\" (UniqueName: \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\") pod \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\" (UID: \"45682fdf-06d1-476e-8e5d-750cf4a32b4e\") " Oct 30 19:35:14.459775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.459730 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca" pod "45682fdf-06d1-476e-8e5d-750cf4a32b4e" (UID: "45682fdf-06d1-476e-8e5d-750cf4a32b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:14.461373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.461338 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca" pod "45682fdf-06d1-476e-8e5d-750cf4a32b4e" (UID: "45682fdf-06d1-476e-8e5d-750cf4a32b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:14.461554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.461534 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm" pod "45682fdf-06d1-476e-8e5d-750cf4a32b4e" (UID: "45682fdf-06d1-476e-8e5d-750cf4a32b4e"). InnerVolumeSpecName "kube-api-access-dr6cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:14.561007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.560932 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\" (UniqueName: \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\") pod \"8c58ce6b-2074-478d-8400-970134c9c081\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " Oct 30 19:35:14.561007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.560965 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\") pod \"8c58ce6b-2074-478d-8400-970134c9c081\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " Oct 30 19:35:14.561195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561020 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\") pod \"8c58ce6b-2074-478d-8400-970134c9c081\" (UID: \"8c58ce6b-2074-478d-8400-970134c9c081\") " Oct 30 19:35:14.561443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561420 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca" pod "8c58ce6b-2074-478d-8400-970134c9c081" (UID: "8c58ce6b-2074-478d-8400-970134c9c081"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:14.561526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561509 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:14.561587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561532 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6cm\" (UniqueName: \"kubernetes.io/projected/45682fdf-06d1-476e-8e5d-750cf4a32b4e-kube-api-access-dr6cm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:14.561587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561546 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:14.561587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.561557 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45682fdf-06d1-476e-8e5d-750cf4a32b4e-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:14.563110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.563087 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca" pod "8c58ce6b-2074-478d-8400-970134c9c081" (UID: "8c58ce6b-2074-478d-8400-970134c9c081"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:14.563178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.563144 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk" pod "8c58ce6b-2074-478d-8400-970134c9c081" (UID: "8c58ce6b-2074-478d-8400-970134c9c081"). InnerVolumeSpecName "kube-api-access-7b4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:14.662400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.662375 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4rk\" (UniqueName: \"kubernetes.io/projected/8c58ce6b-2074-478d-8400-970134c9c081-kube-api-access-7b4rk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:14.662400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:14.662400 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c58ce6b-2074-478d-8400-970134c9c081-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:15.210863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.210821 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" event={"ID":"45682fdf-06d1-476e-8e5d-750cf4a32b4e","Type":"ContainerDied","Data":"2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85"} Oct 30 19:35:15.211038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.210879 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c9ddccd68f2dd753da0233f92c41e1bb62da3c38f49daba8bb986dd01688a85" Oct 30 19:35:15.211038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.210824 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb" Oct 30 19:35:15.212394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.212372 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" Oct 30 19:35:15.212509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.212401 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5" event={"ID":"8c58ce6b-2074-478d-8400-970134c9c081","Type":"ContainerDied","Data":"24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878"} Oct 30 19:35:15.212509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.212447 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d4d6c9e345c8efd5db740d86f4f09526041221a35d750c2e50527be4103878" Oct 30 19:35:15.320844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.320798 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:35:15.321173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.321125 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" containerID="cri-o://51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5" gracePeriod=30 Oct 30 19:35:15.589979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.589888 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49"] Oct 30 19:35:15.596490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.596452 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2a5c13-fbm49"] Oct 30 19:35:15.667928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:15.667893 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7d0528-242e-4b4a-857d-0e19b177bff1" path="/var/lib/kubelet/pods/1b7d0528-242e-4b4a-857d-0e19b177bff1/volumes" Oct 30 19:35:16.257681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:16.257646 4789 patch_prober.go:29] interesting pod/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:35:16.257851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:16.257700 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:35:18.196401 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:18.196366 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:35:19.006487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.006428 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" podUID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" containerName="payload-processing" containerID="cri-o://baeb4d1e94d136a08548d006c9a06e31ec9c14b5cb0c19da7ac8abed29537145" gracePeriod=25 Oct 30 19:35:19.107965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.107896 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" podUID="8bad0e8d-e978-49e7-a823-b9ab84622b17" containerName="payload-pre-processing" containerID="cri-o://0c07e5ba50334a35fa09aac494338a4baecda909c64a2cc3bcbe7cfdb6371c1d" gracePeriod=25 Oct 30 19:35:19.244507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.244472 4789 generic.go:356] "Generic (PLEG): container finished" podID="8bad0e8d-e978-49e7-a823-b9ab84622b17" containerID="0c07e5ba50334a35fa09aac494338a4baecda909c64a2cc3bcbe7cfdb6371c1d" exitCode=0 Oct 30 19:35:19.244915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.244542 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" event={"ID":"8bad0e8d-e978-49e7-a823-b9ab84622b17","Type":"ContainerDied","Data":"0c07e5ba50334a35fa09aac494338a4baecda909c64a2cc3bcbe7cfdb6371c1d"} Oct 30 19:35:19.246886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.246728 4789 generic.go:356] "Generic (PLEG): container finished" podID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" containerID="baeb4d1e94d136a08548d006c9a06e31ec9c14b5cb0c19da7ac8abed29537145" exitCode=0 Oct 30 19:35:19.246886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.246817 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" event={"ID":"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043","Type":"ContainerDied","Data":"baeb4d1e94d136a08548d006c9a06e31ec9c14b5cb0c19da7ac8abed29537145"} Oct 30 19:35:19.273504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.273437 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:35:19.406044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.406013 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:35:19.414978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.414949 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\") pod \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " Oct 30 19:35:19.415114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.415088 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\" (UniqueName: \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\") pod \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\" (UID: \"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043\") " Oct 30 19:35:19.415384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.415359 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume" pod "e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" (UID: "e0c6b24e-40b8-4f9d-9315-dbb72ab7a043"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:19.415540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.415521 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:19.417666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.417644 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x" pod "e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" (UID: "e0c6b24e-40b8-4f9d-9315-dbb72ab7a043"). InnerVolumeSpecName "kube-api-access-2489x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:19.516307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.516170 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\") pod \"8bad0e8d-e978-49e7-a823-b9ab84622b17\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " Oct 30 19:35:19.516307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.516285 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\" (UniqueName: \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\") pod \"8bad0e8d-e978-49e7-a823-b9ab84622b17\" (UID: \"8bad0e8d-e978-49e7-a823-b9ab84622b17\") " Oct 30 19:35:19.516646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.516621 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume" pod "8bad0e8d-e978-49e7-a823-b9ab84622b17" (UID: "8bad0e8d-e978-49e7-a823-b9ab84622b17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:19.517784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.516761 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2489x\" (UniqueName: \"kubernetes.io/projected/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043-kube-api-access-2489x\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:19.517784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.516780 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bad0e8d-e978-49e7-a823-b9ab84622b17-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:19.519053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.519026 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp" pod "8bad0e8d-e978-49e7-a823-b9ab84622b17" (UID: "8bad0e8d-e978-49e7-a823-b9ab84622b17"). InnerVolumeSpecName "kube-api-access-t68dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:19.618150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:19.618082 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68dp\" (UniqueName: \"kubernetes.io/projected/8bad0e8d-e978-49e7-a823-b9ab84622b17-kube-api-access-t68dp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.254228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.254192 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" Oct 30 19:35:20.254670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.254258 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww" event={"ID":"e0c6b24e-40b8-4f9d-9315-dbb72ab7a043","Type":"ContainerDied","Data":"cd9c305104a0c61b0a2f10934814b011ebbe4a95996686baadc5cb0387ac8cd0"} Oct 30 19:35:20.254670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.254320 4789 scope.go:122] "RemoveContainer" containerID="baeb4d1e94d136a08548d006c9a06e31ec9c14b5cb0c19da7ac8abed29537145" Oct 30 19:35:20.255799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.255775 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" event={"ID":"8bad0e8d-e978-49e7-a823-b9ab84622b17","Type":"ContainerDied","Data":"5bd003163eeeab2c1c070dcf38938ae90385bf3395d5d106ee9e8467fdc3b3d1"} Oct 30 19:35:20.255885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.255803 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k" Oct 30 19:35:20.264652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.264634 4789 scope.go:122] "RemoveContainer" containerID="0c07e5ba50334a35fa09aac494338a4baecda909c64a2cc3bcbe7cfdb6371c1d" Oct 30 19:35:20.409393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.409370 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:35:20.455892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.455866 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-5b579977c4-kdpww"] Oct 30 19:35:20.518251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.516609 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:35:20.521729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.521691 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-83ac32-56fbb9457f-62h2k"] Oct 30 19:35:20.595411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.595386 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:35:20.727689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727648 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.727852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727764 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.727852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727805 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.727956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727855 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.727956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727892 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.728075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.727987 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.728125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.728074 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.728125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.728110 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.728262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.728158 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\") pod \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\" (UID: \"95ddb744-c12f-463e-912c-92a1b2e0ca4a\") " Oct 30 19:35:20.728914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.728886 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:20.729106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.729086 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:20.730127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.730099 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:20.730337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.730258 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:20.730774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.730751 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:20.732636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.732548 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:20.732738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.732667 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:20.733560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.733534 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "kube-api-access-4qv9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:20.733640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.733572 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token" pod "95ddb744-c12f-463e-912c-92a1b2e0ca4a" (UID: "95ddb744-c12f-463e-912c-92a1b2e0ca4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:20.829672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829610 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829634 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829647 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829660 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829673 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829687 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qv9t\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-kube-api-access-4qv9t\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829700 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829712 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95ddb744-c12f-463e-912c-92a1b2e0ca4a-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:20.829871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:20.829724 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95ddb744-c12f-463e-912c-92a1b2e0ca4a-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:21.263658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.263627 4789 generic.go:356] "Generic (PLEG): container finished" podID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerID="51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5" exitCode=0 Oct 30 19:35:21.264100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.263688 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" Oct 30 19:35:21.264100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.263700 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerDied","Data":"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5"} Oct 30 19:35:21.264100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.263773 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw" event={"ID":"95ddb744-c12f-463e-912c-92a1b2e0ca4a","Type":"ContainerDied","Data":"54d9d749e160458ea6bfcc938dd1e33cb3a19634bb6fbca4c925dbc5487cc5e6"} Oct 30 19:35:21.264100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.263805 4789 scope.go:122] "RemoveContainer" containerID="51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5" Oct 30 19:35:21.274302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.274285 4789 scope.go:122] "RemoveContainer" containerID="77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef" Oct 30 19:35:21.283368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.283352 4789 scope.go:122] "RemoveContainer" containerID="51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5" Oct 30 19:35:21.283629 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:21.283607 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5\": container with ID starting with 51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5 not found: ID does not exist" containerID="51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5" Oct 30 19:35:21.283681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.283636 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5"} err="failed to get container status \"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5\": rpc error: code = NotFound desc = could not find container \"51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5\": container with ID starting with 51460f2f42280ccaaf5d279505b51a788d985fc8ae045b43ad823c31ed535df5 not found: ID does not exist" Oct 30 19:35:21.283681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.283651 4789 scope.go:122] "RemoveContainer" containerID="77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef" Oct 30 19:35:21.283873 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:21.283854 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef\": container with ID starting with 77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef not found: ID does not exist" containerID="77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef" Oct 30 19:35:21.283932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.283882 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef"} err="failed to get container status \"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef\": rpc error: code = NotFound desc = could not find container \"77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef\": container with ID starting with 77ffbb622a89d7fb5715983861ed69bcd4c9acbf1a0946db825e17ff4c53e3ef not found: ID does not exist" Oct 30 19:35:21.420740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.420708 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:35:21.465292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.465265 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e73f1a-openshift-default-6ffd7485fc-zvvdw"] Oct 30 19:35:21.667725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.667662 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bad0e8d-e978-49e7-a823-b9ab84622b17" path="/var/lib/kubelet/pods/8bad0e8d-e978-49e7-a823-b9ab84622b17/volumes" Oct 30 19:35:21.668093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.668080 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" path="/var/lib/kubelet/pods/95ddb744-c12f-463e-912c-92a1b2e0ca4a/volumes" Oct 30 19:35:21.668457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.668444 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" path="/var/lib/kubelet/pods/e0c6b24e-40b8-4f9d-9315-dbb72ab7a043/volumes" Oct 30 19:35:21.801542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.801511 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg"] Oct 30 19:35:21.802139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802119 4789 cpu_manager.go:414] "containerMap: removing container" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:21.802183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802143 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:21.802183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802169 4789 cpu_manager.go:414] "containerMap: removing container" podUID="45682fdf-06d1-476e-8e5d-750cf4a32b4e" containerName="revoke-keys" Oct 30 19:35:21.802183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802178 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45682fdf-06d1-476e-8e5d-750cf4a32b4e" containerName="revoke-keys" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802191 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8bad0e8d-e978-49e7-a823-b9ab84622b17" containerName="payload-pre-processing" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802199 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bad0e8d-e978-49e7-a823-b9ab84622b17" containerName="payload-pre-processing" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802243 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8c58ce6b-2074-478d-8400-970134c9c081" containerName="revoke-keys" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802251 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c58ce6b-2074-478d-8400-970134c9c081" containerName="revoke-keys" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802267 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" containerName="payload-processing" Oct 30 19:35:21.802308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802274 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" containerName="payload-processing" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802413 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="45682fdf-06d1-476e-8e5d-750cf4a32b4e" containerName="revoke-keys" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802428 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bad0e8d-e978-49e7-a823-b9ab84622b17" containerName="payload-pre-processing" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802441 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c58ce6b-2074-478d-8400-970134c9c081" containerName="revoke-keys" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802450 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802457 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c6b24e-40b8-4f9d-9315-dbb72ab7a043" containerName="payload-processing" Oct 30 19:35:21.802489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.802472 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:21.807820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.807802 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:21.857800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.857774 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:21.857800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.857788 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:21.857970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.857773 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:21.857970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.857845 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:35:21.859685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.859664 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg"] Oct 30 19:35:21.942825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.942800 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:21.942945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.942849 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:21.943002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:21.942980 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ksn\" (UniqueName: \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.044154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.044130 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.044305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.044177 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.044305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.044265 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ksn\" (UniqueName: \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.044922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.044902 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.046486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.046469 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.101702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.101679 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ksn\" (UniqueName: \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\") pod \"maas-api-revoke-sub-65d0c0db-n9wpg\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.123477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.123447 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:22.493874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:22.493847 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg"] Oct 30 19:35:22.496911 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:22.496884 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bd2536_fd1c_4345_aa1a_865505c86cd6.slice/crio-d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93 WatchSource:0}: Error finding container d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93: Status 404 returned error can't find the container with id d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93 Oct 30 19:35:23.281409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:23.281379 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" event={"ID":"46bd2536-fd1c-4345-aa1a-865505c86cd6","Type":"ContainerStarted","Data":"3445accc787857a3610331831000542b5c447c0c7b74bffa06ac282aff78f81d"} Oct 30 19:35:23.281714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:23.281433 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" event={"ID":"46bd2536-fd1c-4345-aa1a-865505c86cd6","Type":"ContainerStarted","Data":"d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93"} Oct 30 19:35:23.392006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:23.391955 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" podStartSLOduration=2.391941625 podStartE2EDuration="2.391941625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:23.390093578 +0000 UTC m=+2959.892004089" watchObservedRunningTime="2026-10-30 19:35:23.391941625 +0000 UTC m=+2959.893852147" Oct 30 19:35:23.511880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:23.511834 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:35:23.512680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:23.512645 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" containerID="cri-o://870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7" gracePeriod=30 Oct 30 19:35:24.288604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:24.288568 4789 generic.go:356] "Generic (PLEG): container finished" podID="46bd2536-fd1c-4345-aa1a-865505c86cd6" containerID="3445accc787857a3610331831000542b5c447c0c7b74bffa06ac282aff78f81d" exitCode=0 Oct 30 19:35:24.288803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:24.288632 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" event={"ID":"46bd2536-fd1c-4345-aa1a-865505c86cd6","Type":"ContainerDied","Data":"3445accc787857a3610331831000542b5c447c0c7b74bffa06ac282aff78f81d"} Oct 30 19:35:25.246664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.246631 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn"] Oct 30 19:35:25.247099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.247085 4789 cpu_manager.go:414] "containerMap: removing container" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:25.247146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.247100 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ddb744-c12f-463e-912c-92a1b2e0ca4a" containerName="istio-proxy" Oct 30 19:35:25.250647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.250627 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.344143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.344108 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn"] Oct 30 19:35:25.374874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.374846 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.375007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.374909 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6g4\" (UniqueName: \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.375057 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.375034 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.469419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.469391 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:25.476655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.476633 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.476732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.476695 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6g4\" (UniqueName: \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.476767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.476738 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.477267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.477245 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.479062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.479037 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.535152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.535092 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6g4\" (UniqueName: \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\") pod \"maas-api-revoke-sub-99d9b39e-dj6dn\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.562946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.562925 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:25.578092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.578064 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\" (UniqueName: \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\") pod \"46bd2536-fd1c-4345-aa1a-865505c86cd6\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " Oct 30 19:35:25.578188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.578148 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\") pod \"46bd2536-fd1c-4345-aa1a-865505c86cd6\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " Oct 30 19:35:25.578188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.578179 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\") pod \"46bd2536-fd1c-4345-aa1a-865505c86cd6\" (UID: \"46bd2536-fd1c-4345-aa1a-865505c86cd6\") " Oct 30 19:35:25.578506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.578484 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca" pod "46bd2536-fd1c-4345-aa1a-865505c86cd6" (UID: "46bd2536-fd1c-4345-aa1a-865505c86cd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:25.578683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.578662 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:25.580964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.580940 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca" pod "46bd2536-fd1c-4345-aa1a-865505c86cd6" (UID: "46bd2536-fd1c-4345-aa1a-865505c86cd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:25.581556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.581239 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn" pod "46bd2536-fd1c-4345-aa1a-865505c86cd6" (UID: "46bd2536-fd1c-4345-aa1a-865505c86cd6"). InnerVolumeSpecName "kube-api-access-46ksn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:25.679317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.679290 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ksn\" (UniqueName: \"kubernetes.io/projected/46bd2536-fd1c-4345-aa1a-865505c86cd6-kube-api-access-46ksn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:25.679317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.679315 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bd2536-fd1c-4345-aa1a-865505c86cd6-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:25.941275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:25.941170 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn"] Oct 30 19:35:25.943538 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:25.943507 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6327c370_f397_4af2_baab_7f33b76407f5.slice/crio-2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874 WatchSource:0}: Error finding container 2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874: Status 404 returned error can't find the container with id 2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874 Oct 30 19:35:26.303330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.303304 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" Oct 30 19:35:26.303790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.303298 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg" event={"ID":"46bd2536-fd1c-4345-aa1a-865505c86cd6","Type":"ContainerDied","Data":"d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93"} Oct 30 19:35:26.303790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.303439 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b12ef514351e022150258fd0e9ee3a16b701733ecb7283c97932d73e165e93" Oct 30 19:35:26.305483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.305196 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" event={"ID":"6327c370-f397-4af2-baab-7f33b76407f5","Type":"ContainerStarted","Data":"a2f46efb57de2a5c1d283cc4ecb25f7f20345bb78de0a2c932fd19058043136d"} Oct 30 19:35:26.305483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.305263 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" event={"ID":"6327c370-f397-4af2-baab-7f33b76407f5","Type":"ContainerStarted","Data":"2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874"} Oct 30 19:35:26.418622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:26.418563 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" podStartSLOduration=1.418546077 podStartE2EDuration="1.418546077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:26.415318174 +0000 UTC m=+2962.917228661" watchObservedRunningTime="2026-10-30 19:35:26.418546077 +0000 UTC m=+2962.920456568" Oct 30 19:35:27.313366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:27.313304 4789 generic.go:356] "Generic (PLEG): container finished" podID="6327c370-f397-4af2-baab-7f33b76407f5" containerID="a2f46efb57de2a5c1d283cc4ecb25f7f20345bb78de0a2c932fd19058043136d" exitCode=0 Oct 30 19:35:27.313877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:27.313549 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" event={"ID":"6327c370-f397-4af2-baab-7f33b76407f5","Type":"ContainerDied","Data":"a2f46efb57de2a5c1d283cc4ecb25f7f20345bb78de0a2c932fd19058043136d"} Oct 30 19:35:28.627380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.627313 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:28.749380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.731930 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\" (UniqueName: \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\") pod \"6327c370-f397-4af2-baab-7f33b76407f5\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " Oct 30 19:35:28.749380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.731998 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\") pod \"6327c370-f397-4af2-baab-7f33b76407f5\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " Oct 30 19:35:28.749380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.732068 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\") pod \"6327c370-f397-4af2-baab-7f33b76407f5\" (UID: \"6327c370-f397-4af2-baab-7f33b76407f5\") " Oct 30 19:35:28.749896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.749848 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca" pod "6327c370-f397-4af2-baab-7f33b76407f5" (UID: "6327c370-f397-4af2-baab-7f33b76407f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:28.766875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.766818 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4" pod "6327c370-f397-4af2-baab-7f33b76407f5" (UID: "6327c370-f397-4af2-baab-7f33b76407f5"). InnerVolumeSpecName "kube-api-access-qk6g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:28.767500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.767438 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca" pod "6327c370-f397-4af2-baab-7f33b76407f5" (UID: "6327c370-f397-4af2-baab-7f33b76407f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:28.837996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.829381 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:35:28.837996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.833753 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6g4\" (UniqueName: \"kubernetes.io/projected/6327c370-f397-4af2-baab-7f33b76407f5-kube-api-access-qk6g4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.837996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.833781 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.837996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.833798 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6327c370-f397-4af2-baab-7f33b76407f5-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.934419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934384 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934453 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934550 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934643 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934674 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934725 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934763 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.934906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934805 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:28.934906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934817 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.935069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934902 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\") pod \"ef3882b5-54e1-47ed-8274-9d1f0618f473\" (UID: \"ef3882b5-54e1-47ed-8274-9d1f0618f473\") " Oct 30 19:35:28.935069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934902 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:28.935069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.934921 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:28.935252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935193 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:28.935384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935359 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef3882b5-54e1-47ed-8274-9d1f0618f473-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.935455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935390 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.935455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935406 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.935455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935420 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:28.935455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.935367 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:28.936749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.936719 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:35:28.936851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.936727 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:28.936851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.936787 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "kube-api-access-pbb2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:28.937223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:28.937189 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token" pod "ef3882b5-54e1-47ed-8274-9d1f0618f473" (UID: "ef3882b5-54e1-47ed-8274-9d1f0618f473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:29.037054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.037018 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.037054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.037053 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.037304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.037067 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.037304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.037080 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef3882b5-54e1-47ed-8274-9d1f0618f473-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.037304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.037094 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbb2s\" (UniqueName: \"kubernetes.io/projected/ef3882b5-54e1-47ed-8274-9d1f0618f473-kube-api-access-pbb2s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.303282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303193 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc"] Oct 30 19:35:29.303709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303687 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:29.303709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303706 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303719 4789 cpu_manager.go:414] "containerMap: removing container" podUID="46bd2536-fd1c-4345-aa1a-865505c86cd6" containerName="revoke-keys" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303727 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bd2536-fd1c-4345-aa1a-865505c86cd6" containerName="revoke-keys" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303748 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303756 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303777 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6327c370-f397-4af2-baab-7f33b76407f5" containerName="revoke-keys" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303782 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6327c370-f397-4af2-baab-7f33b76407f5" containerName="revoke-keys" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303872 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6327c370-f397-4af2-baab-7f33b76407f5" containerName="revoke-keys" Oct 30 19:35:29.303891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303886 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:29.304167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.303895 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bd2536-fd1c-4345-aa1a-865505c86cd6" containerName="revoke-keys" Oct 30 19:35:29.307492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.307465 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.332771 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.332746 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerID="870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7" exitCode=0 Oct 30 19:35:29.332901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.332821 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" Oct 30 19:35:29.332901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.332836 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerDied","Data":"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7"} Oct 30 19:35:29.332901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.332889 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2" event={"ID":"ef3882b5-54e1-47ed-8274-9d1f0618f473","Type":"ContainerDied","Data":"bfc72102a62cd10047f018b133557a8fbd51287df6b007dc6e87f59b6eb88e5b"} Oct 30 19:35:29.333080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.332916 4789 scope.go:122] "RemoveContainer" containerID="870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7" Oct 30 19:35:29.334674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.334653 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" Oct 30 19:35:29.334674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.334664 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn" event={"ID":"6327c370-f397-4af2-baab-7f33b76407f5","Type":"ContainerDied","Data":"2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874"} Oct 30 19:35:29.334845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.334707 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2240a1e6371ecd4b261432427ea470b3f6bdb1e137ee4096d5d7793c7683e874" Oct 30 19:35:29.340407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.340385 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbt95\" (UniqueName: \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.340537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.340450 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.340618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.340535 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.343820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.343796 4789 scope.go:122] "RemoveContainer" containerID="06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa" Oct 30 19:35:29.353512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.353496 4789 scope.go:122] "RemoveContainer" containerID="870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7" Oct 30 19:35:29.353842 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:29.353814 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7\": container with ID starting with 870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7 not found: ID does not exist" containerID="870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7" Oct 30 19:35:29.353899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.353854 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7"} err="failed to get container status \"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7\": rpc error: code = NotFound desc = could not find container \"870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7\": container with ID starting with 870c426c2e3d992769613b36080b80582db69552e876faa3a98fd96ed72b78e7 not found: ID does not exist" Oct 30 19:35:29.353899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.353877 4789 scope.go:122] "RemoveContainer" containerID="06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa" Oct 30 19:35:29.354139 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:29.354116 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa\": container with ID starting with 06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa not found: ID does not exist" containerID="06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa" Oct 30 19:35:29.354219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.354145 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa"} err="failed to get container status \"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa\": rpc error: code = NotFound desc = could not find container \"06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa\": container with ID starting with 06a209e5dd07159696ceedc63c8016281e5cd2436b52e7620bf005dcdc5dadfa not found: ID does not exist" Oct 30 19:35:29.363590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.363570 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc"] Oct 30 19:35:29.441041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.441018 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.441166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.441069 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.441166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.441119 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbt95\" (UniqueName: \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.441700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.441677 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.443349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.443329 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.498795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.498767 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbt95\" (UniqueName: \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\") pod \"maas-api-revoke-sub-4756d9bd-hq6bc\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.573814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.573742 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:35:29.576372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.576344 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2a5c13-openshift-default-57c7bd9975-c6hp2"] Oct 30 19:35:29.622110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.622080 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:29.669524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.669492 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" path="/var/lib/kubelet/pods/ef3882b5-54e1-47ed-8274-9d1f0618f473/volumes" Oct 30 19:35:29.996737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:29.996706 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc"] Oct 30 19:35:30.000477 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:30.000448 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37afaa6_a08f_46ca_9081_4b4826696776.slice/crio-d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf WatchSource:0}: Error finding container d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf: Status 404 returned error can't find the container with id d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf Oct 30 19:35:30.343814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:30.343785 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" event={"ID":"e37afaa6-a08f-46ca-9081-4b4826696776","Type":"ContainerStarted","Data":"c7f4607dd52f59810a97f9f61432b716684d6148626cc070d1a160d5d7cdbee8"} Oct 30 19:35:30.343941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:30.343838 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" event={"ID":"e37afaa6-a08f-46ca-9081-4b4826696776","Type":"ContainerStarted","Data":"d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf"} Oct 30 19:35:30.453192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:30.453149 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" podStartSLOduration=1.453136154 podStartE2EDuration="1.453136154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:30.452608992 +0000 UTC m=+2966.954519479" watchObservedRunningTime="2026-10-30 19:35:30.453136154 +0000 UTC m=+2966.955046639" Oct 30 19:35:31.351734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:31.351702 4789 generic.go:356] "Generic (PLEG): container finished" podID="e37afaa6-a08f-46ca-9081-4b4826696776" containerID="c7f4607dd52f59810a97f9f61432b716684d6148626cc070d1a160d5d7cdbee8" exitCode=0 Oct 30 19:35:31.352120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:31.351789 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" event={"ID":"e37afaa6-a08f-46ca-9081-4b4826696776","Type":"ContainerDied","Data":"c7f4607dd52f59810a97f9f61432b716684d6148626cc070d1a160d5d7cdbee8"} Oct 30 19:35:32.489862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.489839 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:32.570044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.570020 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\") pod \"e37afaa6-a08f-46ca-9081-4b4826696776\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " Oct 30 19:35:32.570168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.570117 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\" (UniqueName: \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\") pod \"e37afaa6-a08f-46ca-9081-4b4826696776\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " Oct 30 19:35:32.570168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.570144 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\") pod \"e37afaa6-a08f-46ca-9081-4b4826696776\" (UID: \"e37afaa6-a08f-46ca-9081-4b4826696776\") " Oct 30 19:35:32.570514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.570489 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca" pod "e37afaa6-a08f-46ca-9081-4b4826696776" (UID: "e37afaa6-a08f-46ca-9081-4b4826696776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:32.572109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.572090 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca" pod "e37afaa6-a08f-46ca-9081-4b4826696776" (UID: "e37afaa6-a08f-46ca-9081-4b4826696776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:32.572174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.572101 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95" pod "e37afaa6-a08f-46ca-9081-4b4826696776" (UID: "e37afaa6-a08f-46ca-9081-4b4826696776"). InnerVolumeSpecName "kube-api-access-jbt95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:32.670993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.670933 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbt95\" (UniqueName: \"kubernetes.io/projected/e37afaa6-a08f-46ca-9081-4b4826696776-kube-api-access-jbt95\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:32.670993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.670959 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:32.670993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:32.670970 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e37afaa6-a08f-46ca-9081-4b4826696776-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:33.365480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:33.365448 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" Oct 30 19:35:33.365658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:33.365456 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc" event={"ID":"e37afaa6-a08f-46ca-9081-4b4826696776","Type":"ContainerDied","Data":"d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf"} Oct 30 19:35:33.365658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:33.365602 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e9fd516442d476e55bba3d6c5b9d124ad3cc5f4b3ee3ce96c533b4d3c7edaf" Oct 30 19:35:35.222053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.222016 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:35:35.222599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.222583 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e37afaa6-a08f-46ca-9081-4b4826696776" containerName="revoke-keys" Oct 30 19:35:35.222674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.222601 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37afaa6-a08f-46ca-9081-4b4826696776" containerName="revoke-keys" Oct 30 19:35:35.222739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.222702 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3882b5-54e1-47ed-8274-9d1f0618f473" containerName="istio-proxy" Oct 30 19:35:35.222739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.222714 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37afaa6-a08f-46ca-9081-4b4826696776" containerName="revoke-keys" Oct 30 19:35:35.226192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.226165 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.298640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.298606 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tz8s\" (UniqueName: \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.298824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.298759 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.321693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.321661 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-83ac32\"" Oct 30 19:35:35.321693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.321661 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-83ac32-dockercfg-tfr2j\"" Oct 30 19:35:35.325987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.325958 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:35:35.400406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.400375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tz8s\" (UniqueName: \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.400571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.400445 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.401096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.401076 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.458684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.458654 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tz8s\" (UniqueName: \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\") pod \"payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.540710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.540623 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:35.912425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:35.912397 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:35:35.914238 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:35.914197 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a9041c_2649_417c_97a6_c4acd578bc3a.slice/crio-0f80ff7d4a13080b87f8675458bfb4f7c8963cf7765cbc3db29b6f1f56ca4ec9 WatchSource:0}: Error finding container 0f80ff7d4a13080b87f8675458bfb4f7c8963cf7765cbc3db29b6f1f56ca4ec9: Status 404 returned error can't find the container with id 0f80ff7d4a13080b87f8675458bfb4f7c8963cf7765cbc3db29b6f1f56ca4ec9 Oct 30 19:35:36.387358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:36.387318 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" event={"ID":"f9a9041c-2649-417c-97a6-c4acd578bc3a","Type":"ContainerStarted","Data":"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796"} Oct 30 19:35:36.387809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:36.387501 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:36.387809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:36.387528 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" event={"ID":"f9a9041c-2649-417c-97a6-c4acd578bc3a","Type":"ContainerStarted","Data":"0f80ff7d4a13080b87f8675458bfb4f7c8963cf7765cbc3db29b6f1f56ca4ec9"} Oct 30 19:35:36.500204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:36.500140 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" podStartSLOduration=1.500121682 podStartE2EDuration="1.500121682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:36.497909642 +0000 UTC m=+2972.999820127" watchObservedRunningTime="2026-10-30 19:35:36.500121682 +0000 UTC m=+2973.002032170" Oct 30 19:35:39.246169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.246126 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98"] Oct 30 19:35:39.250153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.250130 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.301744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.301716 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:39.301744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.301732 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:39.344439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.344397 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.344643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.344446 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.344720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.344642 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zth\" (UniqueName: \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.347881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.347862 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:39.348095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.348073 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:35:39.350871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.350851 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98"] Oct 30 19:35:39.445465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.445429 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zth\" (UniqueName: \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.445651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.445585 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.445651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.445605 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.446303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.446278 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.447946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.447920 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.503776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.503704 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zth\" (UniqueName: \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\") pod \"maas-api-revoke-sub-3444fd80-ckn98\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.563129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.563098 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:39.936642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:39.936610 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98"] Oct 30 19:35:39.938156 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:39.938130 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39fa37d_b715_47ea_9bef_5df2251a01ab.slice/crio-b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970 WatchSource:0}: Error finding container b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970: Status 404 returned error can't find the container with id b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970 Oct 30 19:35:40.416959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:40.416916 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" event={"ID":"b39fa37d-b715-47ea-9bef-5df2251a01ab","Type":"ContainerStarted","Data":"37ff196a9e3c255c4db500b6011c5927d65b0bc7a5f1dd9df22505ee04f164d6"} Oct 30 19:35:40.417334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:40.416975 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" event={"ID":"b39fa37d-b715-47ea-9bef-5df2251a01ab","Type":"ContainerStarted","Data":"b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970"} Oct 30 19:35:40.527445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:40.527331 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" podStartSLOduration=1.527311606 podStartE2EDuration="1.527311606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:40.52622128 +0000 UTC m=+2977.028131770" watchObservedRunningTime="2026-10-30 19:35:40.527311606 +0000 UTC m=+2977.029222093" Oct 30 19:35:41.424477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:41.424437 4789 generic.go:356] "Generic (PLEG): container finished" podID="b39fa37d-b715-47ea-9bef-5df2251a01ab" containerID="37ff196a9e3c255c4db500b6011c5927d65b0bc7a5f1dd9df22505ee04f164d6" exitCode=0 Oct 30 19:35:41.424914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:41.424513 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" event={"ID":"b39fa37d-b715-47ea-9bef-5df2251a01ab","Type":"ContainerDied","Data":"37ff196a9e3c255c4db500b6011c5927d65b0bc7a5f1dd9df22505ee04f164d6"} Oct 30 19:35:42.580403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.580373 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:42.678182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.678152 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\" (UniqueName: \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\") pod \"b39fa37d-b715-47ea-9bef-5df2251a01ab\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " Oct 30 19:35:42.678399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.678259 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\") pod \"b39fa37d-b715-47ea-9bef-5df2251a01ab\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " Oct 30 19:35:42.678399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.678307 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\") pod \"b39fa37d-b715-47ea-9bef-5df2251a01ab\" (UID: \"b39fa37d-b715-47ea-9bef-5df2251a01ab\") " Oct 30 19:35:42.678671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.678642 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca" pod "b39fa37d-b715-47ea-9bef-5df2251a01ab" (UID: "b39fa37d-b715-47ea-9bef-5df2251a01ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:42.680346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.680323 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca" pod "b39fa37d-b715-47ea-9bef-5df2251a01ab" (UID: "b39fa37d-b715-47ea-9bef-5df2251a01ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:42.680427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.680325 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth" pod "b39fa37d-b715-47ea-9bef-5df2251a01ab" (UID: "b39fa37d-b715-47ea-9bef-5df2251a01ab"). InnerVolumeSpecName "kube-api-access-s8zth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:42.780164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.780080 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zth\" (UniqueName: \"kubernetes.io/projected/b39fa37d-b715-47ea-9bef-5df2251a01ab-kube-api-access-s8zth\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:42.780164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.780104 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:42.780164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:42.780113 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39fa37d-b715-47ea-9bef-5df2251a01ab-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:43.448232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:43.448176 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" Oct 30 19:35:43.448232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:43.448193 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98" event={"ID":"b39fa37d-b715-47ea-9bef-5df2251a01ab","Type":"ContainerDied","Data":"b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970"} Oct 30 19:35:43.448455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:43.448280 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5deec045154cd6697cdaf123e9c16555e141372b17a92a628275566dcae1970" Oct 30 19:35:47.396614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:47.396579 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:35:48.196643 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:35:48.196603 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:35:52.125574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.125540 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:35:52.126034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.126017 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b39fa37d-b715-47ea-9bef-5df2251a01ab" containerName="revoke-keys" Oct 30 19:35:52.126034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.126034 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39fa37d-b715-47ea-9bef-5df2251a01ab" containerName="revoke-keys" Oct 30 19:35:52.126170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.126127 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39fa37d-b715-47ea-9bef-5df2251a01ab" containerName="revoke-keys" Oct 30 19:35:52.129758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.129737 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279257 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxpd\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279302 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279388 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279444 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279476 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279496 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279640 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279684 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.279782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.279701 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.329179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.329147 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:35:52.380839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380774 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxpd\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.380839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380808 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380848 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380889 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380910 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380938 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.380978 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381035 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381061 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381434 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381458 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381567 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381576 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.381656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.381604 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.383339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.383313 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.383454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.383437 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.438821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.438795 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.438940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.438820 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxpd\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\") pod \"e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.444839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.444819 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:52.872335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.872309 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:35:52.873763 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:52.873736 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0370e1f_45da_4c11_b695_80c2db8aca48.slice/crio-1d72dfbb01f493f3a73d020272cac92c0bc1fa4e166605ecbfd477aedb549e14 WatchSource:0}: Error finding container 1d72dfbb01f493f3a73d020272cac92c0bc1fa4e166605ecbfd477aedb549e14: Status 404 returned error can't find the container with id 1d72dfbb01f493f3a73d020272cac92c0bc1fa4e166605ecbfd477aedb549e14 Oct 30 19:35:52.875952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.875922 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:52.876041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.875982 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:52.876041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:52.876014 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:35:53.522462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:53.522417 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" event={"ID":"b0370e1f-45da-4c11-b695-80c2db8aca48","Type":"ContainerStarted","Data":"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7"} Oct 30 19:35:53.522869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:53.522516 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" event={"ID":"b0370e1f-45da-4c11-b695-80c2db8aca48","Type":"ContainerStarted","Data":"1d72dfbb01f493f3a73d020272cac92c0bc1fa4e166605ecbfd477aedb549e14"} Oct 30 19:35:53.685263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:53.685185 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podStartSLOduration=1.68517191 podStartE2EDuration="1.68517191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:53.683485569 +0000 UTC m=+2990.185396046" watchObservedRunningTime="2026-10-30 19:35:53.68517191 +0000 UTC m=+2990.187082395" Oct 30 19:35:54.445870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.445827 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:35:54.447291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.447257 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:54.447426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.447355 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:54.790927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.790838 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp"] Oct 30 19:35:54.813232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.812434 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.828887 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.828856 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.829024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.828890 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.829024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.828917 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pgj\" (UniqueName: \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.848818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.848793 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp"] Oct 30 19:35:54.890940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.890914 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:54.890940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.890931 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:54.890940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.890917 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:54.922528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.922503 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:35:54.929588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.929562 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.929747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.929734 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.929891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.929875 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pgj\" (UniqueName: \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.930404 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.930381 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.932053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.932037 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:54.988844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:54.988819 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pgj\" (UniqueName: \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\") pod \"maas-api-revoke-sub-c9f5cf04-jqhjp\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:55.129812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:55.129723 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:55.446165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:55.446131 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:55.446420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:55.446194 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:55.521808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:55.521117 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp"] Oct 30 19:35:55.537724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:55.537695 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" event={"ID":"dd6bfafb-f405-4508-8bb3-87b850a8b786","Type":"ContainerStarted","Data":"e98c44257f9ade25be83362d5c635b3d9fa4e73558b66b313dc60a14ccf05c9b"} Oct 30 19:35:56.446112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:56.446075 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:56.446554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:56.446130 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:56.545630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:56.545594 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" event={"ID":"dd6bfafb-f405-4508-8bb3-87b850a8b786","Type":"ContainerStarted","Data":"b5acf552a088a59130ee9e53f675e322074ef62bbe5f850cc068d95ce99b32fd"} Oct 30 19:35:56.682936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:56.682885 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" podStartSLOduration=2.682869853 podStartE2EDuration="2.682869853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:56.679861307 +0000 UTC m=+2993.181771793" watchObservedRunningTime="2026-10-30 19:35:56.682869853 +0000 UTC m=+2993.184780338" Oct 30 19:35:57.445463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:57.445430 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:57.445644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:57.445483 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:57.554433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:57.554401 4789 generic.go:356] "Generic (PLEG): container finished" podID="dd6bfafb-f405-4508-8bb3-87b850a8b786" containerID="b5acf552a088a59130ee9e53f675e322074ef62bbe5f850cc068d95ce99b32fd" exitCode=0 Oct 30 19:35:57.554852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:57.554475 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" event={"ID":"dd6bfafb-f405-4508-8bb3-87b850a8b786","Type":"ContainerDied","Data":"b5acf552a088a59130ee9e53f675e322074ef62bbe5f850cc068d95ce99b32fd"} Oct 30 19:35:58.281743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.281711 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr"] Oct 30 19:35:58.285652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.285636 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.342845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.342813 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr"] Oct 30 19:35:58.365644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.365609 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnpx\" (UniqueName: \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.365772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.365660 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.365772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.365681 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.445192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.445158 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:58.445363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.445223 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:58.467055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.467023 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnpx\" (UniqueName: \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.467189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.467080 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.467189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.467108 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.467746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.467725 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.469461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.469439 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.531650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.531619 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnpx\" (UniqueName: \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\") pod \"maas-api-revoke-sub-4c7c1a48-l8mjr\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.624694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.624414 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:35:58.769722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.769697 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:58.872271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.872177 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\" (UniqueName: \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\") pod \"dd6bfafb-f405-4508-8bb3-87b850a8b786\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " Oct 30 19:35:58.872455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.872306 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\") pod \"dd6bfafb-f405-4508-8bb3-87b850a8b786\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " Oct 30 19:35:58.872455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.872344 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\") pod \"dd6bfafb-f405-4508-8bb3-87b850a8b786\" (UID: \"dd6bfafb-f405-4508-8bb3-87b850a8b786\") " Oct 30 19:35:58.872656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.872631 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca" pod "dd6bfafb-f405-4508-8bb3-87b850a8b786" (UID: "dd6bfafb-f405-4508-8bb3-87b850a8b786"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:58.874371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.874343 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca" pod "dd6bfafb-f405-4508-8bb3-87b850a8b786" (UID: "dd6bfafb-f405-4508-8bb3-87b850a8b786"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:58.874371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.874359 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj" pod "dd6bfafb-f405-4508-8bb3-87b850a8b786" (UID: "dd6bfafb-f405-4508-8bb3-87b850a8b786"). InnerVolumeSpecName "kube-api-access-26pgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:58.973763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.973732 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:58.973763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.973763 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd6bfafb-f405-4508-8bb3-87b850a8b786-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:58.973958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:58.973776 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pgj\" (UniqueName: \"kubernetes.io/projected/dd6bfafb-f405-4508-8bb3-87b850a8b786-kube-api-access-26pgj\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:59.059202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.059162 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr"] Oct 30 19:35:59.062197 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:35:59.062169 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4646599_4d53_4750_83bf_e042ed4e7a88.slice/crio-943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511 WatchSource:0}: Error finding container 943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511: Status 404 returned error can't find the container with id 943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511 Oct 30 19:35:59.445849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.445813 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 30 19:35:59.446040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.445912 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 30 19:35:59.570644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.570610 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" event={"ID":"a4646599-4d53-4750-83bf-e042ed4e7a88","Type":"ContainerStarted","Data":"fa074e57d93759ea19cf82416c77b30c51da6aaf2e38130ea279198fca350d6b"} Oct 30 19:35:59.570802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.570672 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" event={"ID":"a4646599-4d53-4750-83bf-e042ed4e7a88","Type":"ContainerStarted","Data":"943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511"} Oct 30 19:35:59.572264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.572233 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" event={"ID":"dd6bfafb-f405-4508-8bb3-87b850a8b786","Type":"ContainerDied","Data":"e98c44257f9ade25be83362d5c635b3d9fa4e73558b66b313dc60a14ccf05c9b"} Oct 30 19:35:59.572372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.572279 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98c44257f9ade25be83362d5c635b3d9fa4e73558b66b313dc60a14ccf05c9b" Oct 30 19:35:59.572372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.572291 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp" Oct 30 19:35:59.682257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:35:59.682193 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" podStartSLOduration=1.6821800919999998 podStartE2EDuration="1.682180092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:59.681190921 +0000 UTC m=+2996.183101408" watchObservedRunningTime="2026-10-30 19:35:59.682180092 +0000 UTC m=+2996.184090578" Oct 30 19:36:00.450173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:00.450134 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:36:00.450611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:00.450595 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:36:00.451178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:00.451163 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:36:00.580946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:00.580913 4789 generic.go:356] "Generic (PLEG): container finished" podID="a4646599-4d53-4750-83bf-e042ed4e7a88" containerID="fa074e57d93759ea19cf82416c77b30c51da6aaf2e38130ea279198fca350d6b" exitCode=0 Oct 30 19:36:00.581110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:00.581002 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" event={"ID":"a4646599-4d53-4750-83bf-e042ed4e7a88","Type":"ContainerDied","Data":"fa074e57d93759ea19cf82416c77b30c51da6aaf2e38130ea279198fca350d6b"} Oct 30 19:36:01.746427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.746400 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:36:01.800134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.800108 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\") pod \"a4646599-4d53-4750-83bf-e042ed4e7a88\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " Oct 30 19:36:01.800293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.800155 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\" (UniqueName: \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\") pod \"a4646599-4d53-4750-83bf-e042ed4e7a88\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " Oct 30 19:36:01.800367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.800313 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\") pod \"a4646599-4d53-4750-83bf-e042ed4e7a88\" (UID: \"a4646599-4d53-4750-83bf-e042ed4e7a88\") " Oct 30 19:36:01.800700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.800671 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca" pod "a4646599-4d53-4750-83bf-e042ed4e7a88" (UID: "a4646599-4d53-4750-83bf-e042ed4e7a88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:01.802233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.802179 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca" pod "a4646599-4d53-4750-83bf-e042ed4e7a88" (UID: "a4646599-4d53-4750-83bf-e042ed4e7a88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:01.802367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.802347 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx" pod "a4646599-4d53-4750-83bf-e042ed4e7a88" (UID: "a4646599-4d53-4750-83bf-e042ed4e7a88"). InnerVolumeSpecName "kube-api-access-4cnpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:01.901595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.901512 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:01.901595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.901544 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnpx\" (UniqueName: \"kubernetes.io/projected/a4646599-4d53-4750-83bf-e042ed4e7a88-kube-api-access-4cnpx\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:01.901595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:01.901561 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4646599-4d53-4750-83bf-e042ed4e7a88-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:02.546639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.546603 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5"] Oct 30 19:36:02.547101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547087 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a4646599-4d53-4750-83bf-e042ed4e7a88" containerName="revoke-keys" Oct 30 19:36:02.547154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547103 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4646599-4d53-4750-83bf-e042ed4e7a88" containerName="revoke-keys" Oct 30 19:36:02.547154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547128 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dd6bfafb-f405-4508-8bb3-87b850a8b786" containerName="revoke-keys" Oct 30 19:36:02.547154 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547132 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6bfafb-f405-4508-8bb3-87b850a8b786" containerName="revoke-keys" Oct 30 19:36:02.547307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547221 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4646599-4d53-4750-83bf-e042ed4e7a88" containerName="revoke-keys" Oct 30 19:36:02.547307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.547235 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6bfafb-f405-4508-8bb3-87b850a8b786" containerName="revoke-keys" Oct 30 19:36:02.550362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.550343 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.608670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.608630 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5"] Oct 30 19:36:02.609272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.609247 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.609491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.609470 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.609833 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.609684 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgdg\" (UniqueName: \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.613378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.613344 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" event={"ID":"a4646599-4d53-4750-83bf-e042ed4e7a88","Type":"ContainerDied","Data":"943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511"} Oct 30 19:36:02.613491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.613412 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943580e73606d583b20281f6a4edc964ffaf2e87f2a1b0f169202d898f25c511" Oct 30 19:36:02.613555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.613536 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr" Oct 30 19:36:02.710902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.710857 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.711078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.710947 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.711633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.711269 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgdg\" (UniqueName: \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.712089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.711862 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.713549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.713521 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.768788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.768763 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgdg\" (UniqueName: \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\") pod \"maas-api-revoke-sub-c1c7991c-hskb5\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:02.863982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:02.863916 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:03.236685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:03.236361 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5"] Oct 30 19:36:03.238297 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:03.238250 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ffccdc_1087_478e_80fd_7077f201326d.slice/crio-7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a WatchSource:0}: Error finding container 7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a: Status 404 returned error can't find the container with id 7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a Oct 30 19:36:03.622436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:03.622394 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" event={"ID":"51ffccdc-1087-478e-80fd-7077f201326d","Type":"ContainerStarted","Data":"aba5b21b984fc0734c11d53bfe7a76442b5bd5669f5346e24ca8eba65e694b82"} Oct 30 19:36:03.622621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:03.622458 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" event={"ID":"51ffccdc-1087-478e-80fd-7077f201326d","Type":"ContainerStarted","Data":"7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a"} Oct 30 19:36:03.732849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:03.732796 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" podStartSLOduration=1.7327816999999999 podStartE2EDuration="1.7327817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:03.731436187 +0000 UTC m=+3000.233346674" watchObservedRunningTime="2026-10-30 19:36:03.7327817 +0000 UTC m=+3000.234692186" Oct 30 19:36:04.411362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:04.411272 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:36:04.428237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:04.428194 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:36:04.634229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:04.634099 4789 generic.go:356] "Generic (PLEG): container finished" podID="51ffccdc-1087-478e-80fd-7077f201326d" containerID="aba5b21b984fc0734c11d53bfe7a76442b5bd5669f5346e24ca8eba65e694b82" exitCode=0 Oct 30 19:36:04.634229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:04.634147 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" event={"ID":"51ffccdc-1087-478e-80fd-7077f201326d","Type":"ContainerDied","Data":"aba5b21b984fc0734c11d53bfe7a76442b5bd5669f5346e24ca8eba65e694b82"} Oct 30 19:36:04.747087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:04.747045 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7_f9a9041c-2649-417c-97a6-c4acd578bc3a/payload-processing/0.log" Oct 30 19:36:06.028573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.028543 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:06.086534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.086504 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\" (UniqueName: \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\") pod \"51ffccdc-1087-478e-80fd-7077f201326d\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " Oct 30 19:36:06.086534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.086540 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\") pod \"51ffccdc-1087-478e-80fd-7077f201326d\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " Oct 30 19:36:06.086757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.086590 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\") pod \"51ffccdc-1087-478e-80fd-7077f201326d\" (UID: \"51ffccdc-1087-478e-80fd-7077f201326d\") " Oct 30 19:36:06.086998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.086966 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca" pod "51ffccdc-1087-478e-80fd-7077f201326d" (UID: "51ffccdc-1087-478e-80fd-7077f201326d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:06.088624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.088593 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg" pod "51ffccdc-1087-478e-80fd-7077f201326d" (UID: "51ffccdc-1087-478e-80fd-7077f201326d"). InnerVolumeSpecName "kube-api-access-kpgdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:06.088624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.088609 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca" pod "51ffccdc-1087-478e-80fd-7077f201326d" (UID: "51ffccdc-1087-478e-80fd-7077f201326d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:06.188021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.187986 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpgdg\" (UniqueName: \"kubernetes.io/projected/51ffccdc-1087-478e-80fd-7077f201326d-kube-api-access-kpgdg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:06.188021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.188015 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:06.188253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.188030 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51ffccdc-1087-478e-80fd-7077f201326d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:06.670950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.670866 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" Oct 30 19:36:06.670950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.670904 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5" event={"ID":"51ffccdc-1087-478e-80fd-7077f201326d","Type":"ContainerDied","Data":"7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a"} Oct 30 19:36:06.670950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:06.670948 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7971c6b29bec6e1497713c72c2d935199535a72e9cbdb785659d398a4a0dda9a" Oct 30 19:36:07.283156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.283124 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh"] Oct 30 19:36:07.283703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.283674 4789 cpu_manager.go:414] "containerMap: removing container" podUID="51ffccdc-1087-478e-80fd-7077f201326d" containerName="revoke-keys" Oct 30 19:36:07.283703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.283695 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ffccdc-1087-478e-80fd-7077f201326d" containerName="revoke-keys" Oct 30 19:36:07.283912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.283813 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ffccdc-1087-478e-80fd-7077f201326d" containerName="revoke-keys" Oct 30 19:36:07.288898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.288878 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.296532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.296510 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.296649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.296545 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.296649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.296633 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt7s\" (UniqueName: \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.340503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.340478 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:07.340623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.340576 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:07.340694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.340628 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:36:07.341021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.341000 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:07.344239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.344205 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh"] Oct 30 19:36:07.397703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.397670 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.397851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.397714 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.397851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.397794 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt7s\" (UniqueName: \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.398463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.398445 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.400513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.400490 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.434848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.434818 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj"] Oct 30 19:36:07.438750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.438732 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.456109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.456081 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt7s\" (UniqueName: \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\") pod \"maas-api-revoke-sub-1517eb93-rnflh\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.494601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.494576 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj"] Oct 30 19:36:07.498478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.498444 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.498602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.498504 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.498677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.498617 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtd9\" (UniqueName: \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.599264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.599181 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.599264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.599232 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.599428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.599279 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtd9\" (UniqueName: \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.599979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.599950 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.601571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.601546 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.603523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.603508 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:07.657739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.657712 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtd9\" (UniqueName: \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\") pod \"maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.755178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.755148 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:07.985018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:07.984991 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh"] Oct 30 19:36:07.987297 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:07.987266 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20770d7_a16a_48f9_8416_64ec1af203a1.slice/crio-46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a WatchSource:0}: Error finding container 46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a: Status 404 returned error can't find the container with id 46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a Oct 30 19:36:08.135760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.135734 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj"] Oct 30 19:36:08.136705 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:08.136683 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb64b6f3_be61_4e0e_ad85_68ca19514753.slice/crio-05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864 WatchSource:0}: Error finding container 05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864: Status 404 returned error can't find the container with id 05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864 Oct 30 19:36:08.689367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.689328 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" event={"ID":"bb64b6f3-be61-4e0e-ad85-68ca19514753","Type":"ContainerStarted","Data":"71df29559153f8e1a60851f02b7fa0d15e4b77080f51d8c02a5195cda79f71fb"} Oct 30 19:36:08.689809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.689388 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" event={"ID":"bb64b6f3-be61-4e0e-ad85-68ca19514753","Type":"ContainerStarted","Data":"05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864"} Oct 30 19:36:08.690967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.690947 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" event={"ID":"a20770d7-a16a-48f9-8416-64ec1af203a1","Type":"ContainerStarted","Data":"77ed668160beba18abbc239531da47764b134b43383d1453d6d0c18c7a8719f0"} Oct 30 19:36:08.691031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.690986 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" event={"ID":"a20770d7-a16a-48f9-8416-64ec1af203a1","Type":"ContainerStarted","Data":"46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a"} Oct 30 19:36:08.803492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.803430 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" podStartSLOduration=1.803416624 podStartE2EDuration="1.803416624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:08.801242614 +0000 UTC m=+3005.303153098" watchObservedRunningTime="2026-10-30 19:36:08.803416624 +0000 UTC m=+3005.305327109" Oct 30 19:36:08.915197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:08.915145 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" podStartSLOduration=1.915127019 podStartE2EDuration="1.915127019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:08.913116407 +0000 UTC m=+3005.415026895" watchObservedRunningTime="2026-10-30 19:36:08.915127019 +0000 UTC m=+3005.417037499" Oct 30 19:36:09.698602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:09.698568 4789 generic.go:356] "Generic (PLEG): container finished" podID="bb64b6f3-be61-4e0e-ad85-68ca19514753" containerID="71df29559153f8e1a60851f02b7fa0d15e4b77080f51d8c02a5195cda79f71fb" exitCode=0 Oct 30 19:36:09.699059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:09.698653 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" event={"ID":"bb64b6f3-be61-4e0e-ad85-68ca19514753","Type":"ContainerDied","Data":"71df29559153f8e1a60851f02b7fa0d15e4b77080f51d8c02a5195cda79f71fb"} Oct 30 19:36:09.700253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:09.700200 4789 generic.go:356] "Generic (PLEG): container finished" podID="a20770d7-a16a-48f9-8416-64ec1af203a1" containerID="77ed668160beba18abbc239531da47764b134b43383d1453d6d0c18c7a8719f0" exitCode=0 Oct 30 19:36:09.700377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:09.700303 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" event={"ID":"a20770d7-a16a-48f9-8416-64ec1af203a1","Type":"ContainerDied","Data":"77ed668160beba18abbc239531da47764b134b43383d1453d6d0c18c7a8719f0"} Oct 30 19:36:10.880244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.880196 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:10.883321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.883302 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:10.936661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936635 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\") pod \"a20770d7-a16a-48f9-8416-64ec1af203a1\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " Oct 30 19:36:10.936776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936691 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\") pod \"bb64b6f3-be61-4e0e-ad85-68ca19514753\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " Oct 30 19:36:10.936841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936823 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\") pod \"a20770d7-a16a-48f9-8416-64ec1af203a1\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " Oct 30 19:36:10.936907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936858 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\") pod \"bb64b6f3-be61-4e0e-ad85-68ca19514753\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " Oct 30 19:36:10.936907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936896 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\" (UniqueName: \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\") pod \"a20770d7-a16a-48f9-8416-64ec1af203a1\" (UID: \"a20770d7-a16a-48f9-8416-64ec1af203a1\") " Oct 30 19:36:10.937007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.936940 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\" (UniqueName: \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\") pod \"bb64b6f3-be61-4e0e-ad85-68ca19514753\" (UID: \"bb64b6f3-be61-4e0e-ad85-68ca19514753\") " Oct 30 19:36:10.937118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.937042 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca" pod "bb64b6f3-be61-4e0e-ad85-68ca19514753" (UID: "bb64b6f3-be61-4e0e-ad85-68ca19514753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:10.937249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.937193 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca" pod "a20770d7-a16a-48f9-8416-64ec1af203a1" (UID: "a20770d7-a16a-48f9-8416-64ec1af203a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:10.937401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.937380 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:10.937473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.937406 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:10.938812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.938785 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca" pod "a20770d7-a16a-48f9-8416-64ec1af203a1" (UID: "a20770d7-a16a-48f9-8416-64ec1af203a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:10.938973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.938942 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca" pod "bb64b6f3-be61-4e0e-ad85-68ca19514753" (UID: "bb64b6f3-be61-4e0e-ad85-68ca19514753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:10.939061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.939006 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9" pod "bb64b6f3-be61-4e0e-ad85-68ca19514753" (UID: "bb64b6f3-be61-4e0e-ad85-68ca19514753"). InnerVolumeSpecName "kube-api-access-hjtd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:10.939061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:10.939037 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s" pod "a20770d7-a16a-48f9-8416-64ec1af203a1" (UID: "a20770d7-a16a-48f9-8416-64ec1af203a1"). InnerVolumeSpecName "kube-api-access-vjt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:11.037996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.037914 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb64b6f3-be61-4e0e-ad85-68ca19514753-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:11.037996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.037943 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjt7s\" (UniqueName: \"kubernetes.io/projected/a20770d7-a16a-48f9-8416-64ec1af203a1-kube-api-access-vjt7s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:11.037996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.037954 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtd9\" (UniqueName: \"kubernetes.io/projected/bb64b6f3-be61-4e0e-ad85-68ca19514753-kube-api-access-hjtd9\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:11.037996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.037963 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20770d7-a16a-48f9-8416-64ec1af203a1-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:11.714694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.714665 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" Oct 30 19:36:11.714835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.714700 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj" event={"ID":"bb64b6f3-be61-4e0e-ad85-68ca19514753","Type":"ContainerDied","Data":"05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864"} Oct 30 19:36:11.714835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.714755 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c99dbab3a52e0f80e8f44d32e0cfd51bb6996935b2beb18eb20d349f27c864" Oct 30 19:36:11.716184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.716164 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" Oct 30 19:36:11.716338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.716178 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh" event={"ID":"a20770d7-a16a-48f9-8416-64ec1af203a1","Type":"ContainerDied","Data":"46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a"} Oct 30 19:36:11.716338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:11.716248 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46b0b78dd68b1dd9c6d6b23131c87bb757c04b9961b329f7ce00c980892b612a" Oct 30 19:36:17.228191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.228112 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w"] Oct 30 19:36:17.228911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.228888 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bb64b6f3-be61-4e0e-ad85-68ca19514753" containerName="revoke-keys" Oct 30 19:36:17.228960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.228916 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb64b6f3-be61-4e0e-ad85-68ca19514753" containerName="revoke-keys" Oct 30 19:36:17.228960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.228934 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a20770d7-a16a-48f9-8416-64ec1af203a1" containerName="revoke-keys" Oct 30 19:36:17.228960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.228943 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20770d7-a16a-48f9-8416-64ec1af203a1" containerName="revoke-keys" Oct 30 19:36:17.229084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.229071 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb64b6f3-be61-4e0e-ad85-68ca19514753" containerName="revoke-keys" Oct 30 19:36:17.229117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.229091 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20770d7-a16a-48f9-8416-64ec1af203a1" containerName="revoke-keys" Oct 30 19:36:17.234197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.234176 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.286996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.286966 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w"] Oct 30 19:36:17.298668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.298642 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.298814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.298689 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nh5\" (UniqueName: \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.298814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.298785 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.328347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.328324 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:17.328627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.328610 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:17.328723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.328615 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:17.328723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.328616 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:36:17.399638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.399612 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.399811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.399648 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nh5\" (UniqueName: \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.399811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.399681 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.400501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.400478 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.402137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.402114 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.457945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.457919 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nh5\" (UniqueName: \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\") pod \"maas-api-revoke-sub-2b41acf4-j296w\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.550345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.550292 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:17.925921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:17.925891 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w"] Oct 30 19:36:17.929315 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:17.928583 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2ddc4c_1ea3_4b91_bc81_0db58328b4f1.slice/crio-4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0 WatchSource:0}: Error finding container 4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0: Status 404 returned error can't find the container with id 4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0 Oct 30 19:36:18.196233 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:36:18.196174 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:36:18.767904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:18.767876 4789 generic.go:356] "Generic (PLEG): container finished" podID="ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" containerID="77462bf5c1f74444506daa85809e2bc7485138b0ccaddf880a1f9df33a61e46b" exitCode=0 Oct 30 19:36:18.768277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:18.767954 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" event={"ID":"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1","Type":"ContainerDied","Data":"77462bf5c1f74444506daa85809e2bc7485138b0ccaddf880a1f9df33a61e46b"} Oct 30 19:36:18.768277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:18.768001 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" event={"ID":"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1","Type":"ContainerStarted","Data":"4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0"} Oct 30 19:36:19.926977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:19.926948 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:20.025400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.025367 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\" (UniqueName: \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\") pod \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " Oct 30 19:36:20.025568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.025419 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\") pod \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " Oct 30 19:36:20.025568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.025531 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\") pod \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\" (UID: \"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1\") " Oct 30 19:36:20.025973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.025910 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca" pod "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" (UID: "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:20.027491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.027461 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca" pod "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" (UID: "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:20.027679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.027658 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5" pod "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" (UID: "ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1"). InnerVolumeSpecName "kube-api-access-v7nh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:20.127375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.127350 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nh5\" (UniqueName: \"kubernetes.io/projected/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-kube-api-access-v7nh5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:20.127375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.127374 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:20.127536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.127385 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:20.785407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.785375 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" event={"ID":"ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1","Type":"ContainerDied","Data":"4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0"} Oct 30 19:36:20.785557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.785427 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd2dfb240662f203979d1193382bf6f64c9cae1b5145201a2a325a41cc13cc0" Oct 30 19:36:20.785557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:20.785386 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w" Oct 30 19:36:27.235338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.235298 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:36:27.236126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.236094 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" containerName="revoke-keys" Oct 30 19:36:27.236126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.236122 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" containerName="revoke-keys" Oct 30 19:36:27.236332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.236255 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" containerName="revoke-keys" Oct 30 19:36:27.240573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.240553 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296619 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296712 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296746 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrn9k\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296775 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296799 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.296941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.296919 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.297032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.297015 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.297068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.297046 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.297116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.297081 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.336422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.336395 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-22fb18-openshift-default-dockercfg-pqpjp\"" Oct 30 19:36:27.339669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.339645 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:36:27.343699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.343681 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.372691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.372668 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:36:27.398320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398295 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398338 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398383 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398438 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398471 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398520 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398558 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398611 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398651 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrn9k\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398724 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398766 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398801 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.398883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398830 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.398883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398858 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.399153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398885 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxt4\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.399153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398927 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.399153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.398957 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.399153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.399005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.399833 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.399810 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.399969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.399899 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.400175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.400157 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.400344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.400303 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.400721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.400658 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.402342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.402318 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.402449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.402350 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.420264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.420240 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:36:27.466513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.466491 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrn9k\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.468375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.468357 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.493808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.493757 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2301cb-openshift-default-dockercfg-r29tl\"" Oct 30 19:36:27.500273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500249 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.500395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500286 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.500395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500318 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.500395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500343 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxt4\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.500395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500379 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500414 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500519 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500549 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500604 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500698 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500851 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500878 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.500906 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.501604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.501418 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.502774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.502753 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.502943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.502920 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.554078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.554057 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:27.557889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.557871 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.558317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.558296 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxt4\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.656509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.656282 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:27.989302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:27.989273 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:36:28.018966 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:28.018934 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa515561_b454_4d60_ab83_a38a8cf3c3b7.slice/crio-bab1cf99dcf6252240a426127e75276ed0c5cdf8c22146e1f42f1061cc366bcc WatchSource:0}: Error finding container bab1cf99dcf6252240a426127e75276ed0c5cdf8c22146e1f42f1061cc366bcc: Status 404 returned error can't find the container with id bab1cf99dcf6252240a426127e75276ed0c5cdf8c22146e1f42f1061cc366bcc Oct 30 19:36:28.021500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.021461 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.021636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.021542 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.021636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.021584 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.039686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.039660 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:36:28.040067 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:28.040037 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1ab8e8_001c_4080_a741_fc88b4ce6fbf.slice/crio-99d0b4ed1c097fe3ba662b6c2bfe6a5d98d233a18414c26ed88718b0cbade5f0 WatchSource:0}: Error finding container 99d0b4ed1c097fe3ba662b6c2bfe6a5d98d233a18414c26ed88718b0cbade5f0: Status 404 returned error can't find the container with id 99d0b4ed1c097fe3ba662b6c2bfe6a5d98d233a18414c26ed88718b0cbade5f0 Oct 30 19:36:28.041628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.041601 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.041734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.041662 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.041734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.041687 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:36:28.856155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.856117 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" event={"ID":"fa515561-b454-4d60-ab83-a38a8cf3c3b7","Type":"ContainerStarted","Data":"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea"} Oct 30 19:36:28.856926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.856887 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" event={"ID":"fa515561-b454-4d60-ab83-a38a8cf3c3b7","Type":"ContainerStarted","Data":"bab1cf99dcf6252240a426127e75276ed0c5cdf8c22146e1f42f1061cc366bcc"} Oct 30 19:36:28.860311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.860248 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" event={"ID":"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf","Type":"ContainerStarted","Data":"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178"} Oct 30 19:36:28.860311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.860302 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" event={"ID":"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf","Type":"ContainerStarted","Data":"99d0b4ed1c097fe3ba662b6c2bfe6a5d98d233a18414c26ed88718b0cbade5f0"} Oct 30 19:36:28.979498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:28.979441 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podStartSLOduration=1.97942773 podStartE2EDuration="1.97942773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.976259945 +0000 UTC m=+3025.478170431" watchObservedRunningTime="2026-10-30 19:36:28.97942773 +0000 UTC m=+3025.481338216" Oct 30 19:36:29.132539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.132406 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podStartSLOduration=2.132382272 podStartE2EDuration="2.132382272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:29.112236145 +0000 UTC m=+3025.614146632" watchObservedRunningTime="2026-10-30 19:36:29.132382272 +0000 UTC m=+3025.634292758" Oct 30 19:36:29.555117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.555070 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:29.556756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.556727 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:29.556899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.556780 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:29.656959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.656905 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:29.658850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.658819 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:29.659003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:29.658878 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:30.555689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:30.555652 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:30.556239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:30.555733 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:30.658245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:30.657658 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:30.658245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:30.657739 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:31.529233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.528452 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:36:31.529233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.528701 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" podUID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" containerName="payload-processing" containerID="cri-o://e43f270beee1b6f16f7e2ec8b40f45b1499388a866738551586dff89e37eadb7" gracePeriod=30 Oct 30 19:36:31.558950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.558911 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:31.559491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.559432 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:31.682270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.678149 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:31.702333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.702292 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:31.882455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.882376 4789 generic.go:356] "Generic (PLEG): container finished" podID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" containerID="e43f270beee1b6f16f7e2ec8b40f45b1499388a866738551586dff89e37eadb7" exitCode=0 Oct 30 19:36:31.882599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:31.882446 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" event={"ID":"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2","Type":"ContainerDied","Data":"e43f270beee1b6f16f7e2ec8b40f45b1499388a866738551586dff89e37eadb7"} Oct 30 19:36:32.171505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.171479 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:36:32.224231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.220700 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\" (UniqueName: \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\") pod \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " Oct 30 19:36:32.224231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.220853 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\") pod \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\" (UID: \"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2\") " Oct 30 19:36:32.247233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.238417 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume" pod "353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" (UID: "353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:32.247523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.247396 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72" pod "353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" (UID: "353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2"). InnerVolumeSpecName "kube-api-access-4cd72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:32.344231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.337939 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cd72\" (UniqueName: \"kubernetes.io/projected/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-kube-api-access-4cd72\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:32.344231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.337982 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:32.555321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.555286 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:32.555507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.555337 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:32.657390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.657355 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:32.657878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.657405 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:32.915287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.908764 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" event={"ID":"353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2","Type":"ContainerDied","Data":"86d0370d258a31ab60adc617c52444749eec4e90ac339579928ad594cb79c526"} Oct 30 19:36:32.915287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.908836 4789 scope.go:122] "RemoveContainer" containerID="e43f270beee1b6f16f7e2ec8b40f45b1499388a866738551586dff89e37eadb7" Oct 30 19:36:32.915287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:32.908986 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x" Oct 30 19:36:33.100414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.100380 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:36:33.114223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.114170 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f3b5af-f9b966968-6z64x"] Oct 30 19:36:33.318551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.318519 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj"] Oct 30 19:36:33.333278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.333250 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f3b5af-6hxfj"] Oct 30 19:36:33.563620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.563585 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:33.563898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.563867 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:33.662653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.662568 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:33.700339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.700295 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:33.709046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.709013 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" path="/var/lib/kubelet/pods/353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2/volumes" Oct 30 19:36:33.721232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:33.710041 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb64b6f3-be61-4e0e-ad85-68ca19514753" path="/var/lib/kubelet/pods/bb64b6f3-be61-4e0e-ad85-68ca19514753/volumes" Oct 30 19:36:34.051406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:34.050284 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:36:34.556721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:34.556686 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:36:34.557110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:34.557057 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:36:34.658233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:34.657734 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 19:36:34.658233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:34.657887 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 19:36:35.878063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.878025 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:35.878653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.878630 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:35.879499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.879468 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:36:35.983367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.983338 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:35.983747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.983721 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:35.984723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:35.984699 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:36:40.706076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.706046 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx"] Oct 30 19:36:40.706517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.706503 4789 cpu_manager.go:414] "containerMap: removing container" podUID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" containerName="payload-processing" Oct 30 19:36:40.706573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.706518 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" containerName="payload-processing" Oct 30 19:36:40.706634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.706624 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="353d9a4a-0ee3-46d1-9f8a-7ae79eb10fc2" containerName="payload-processing" Oct 30 19:36:40.711370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.711353 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.763909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.763888 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:40.764037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.763939 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:36:40.793904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.793877 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fcd\" (UniqueName: \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.794031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.793953 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.794120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.794095 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.809970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.809952 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:40.810315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.810296 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:40.813707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.813690 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx"] Oct 30 19:36:40.894742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.894712 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.894898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.894770 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.894967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.894929 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fcd\" (UniqueName: \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.895520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.895498 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.897629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.897600 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:40.953454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:40.953430 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fcd\" (UniqueName: \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\") pod \"maas-api-revoke-sub-d9a26a37-qsxxx\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:41.024390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.024319 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:41.146785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.146750 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799"] Oct 30 19:36:41.150803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.150783 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.204866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.204837 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.205020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.204998 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmv5f\" (UniqueName: \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.205150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.205127 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.208155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.208124 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799"] Oct 30 19:36:41.305809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.305741 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.305960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.305815 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.305960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.305874 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmv5f\" (UniqueName: \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.306518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.306496 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.308102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.308085 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.368731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.368706 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmv5f\" (UniqueName: \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\") pod \"maas-api-revoke-sub-7b4284b7-t8799\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.440234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.437147 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx"] Oct 30 19:36:41.447719 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:41.447688 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b99ced_4252_4e9f_b941_73635f87cd13.slice/crio-391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5 WatchSource:0}: Error finding container 391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5: Status 404 returned error can't find the container with id 391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5 Oct 30 19:36:41.463929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.463905 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:41.841504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:41.841475 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799"] Oct 30 19:36:41.843091 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:41.843066 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b189f7_8f15_4b3d_9fde_0d88a566ade4.slice/crio-049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2 WatchSource:0}: Error finding container 049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2: Status 404 returned error can't find the container with id 049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2 Oct 30 19:36:42.000607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:42.000575 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" event={"ID":"30b189f7-8f15-4b3d-9fde-0d88a566ade4","Type":"ContainerStarted","Data":"049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2"} Oct 30 19:36:42.002098 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:42.002073 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" event={"ID":"28b99ced-4252-4e9f-b941-73635f87cd13","Type":"ContainerStarted","Data":"e46b613ea16b3730ecbce602ec8f92f7d04e613950a173c4737682967fd81dcb"} Oct 30 19:36:42.002246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:42.002114 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" event={"ID":"28b99ced-4252-4e9f-b941-73635f87cd13","Type":"ContainerStarted","Data":"391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5"} Oct 30 19:36:42.123274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:42.123194 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" podStartSLOduration=2.12318086 podStartE2EDuration="2.12318086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:42.11802509 +0000 UTC m=+3038.619935576" watchObservedRunningTime="2026-10-30 19:36:42.12318086 +0000 UTC m=+3038.625091365" Oct 30 19:36:43.012049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:43.011965 4789 generic.go:356] "Generic (PLEG): container finished" podID="30b189f7-8f15-4b3d-9fde-0d88a566ade4" containerID="2d424d4fe8ccbdd8ffe8eaf340270903a87a1414f8beef66c4b0310bffd60ac5" exitCode=0 Oct 30 19:36:43.012487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:43.012050 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" event={"ID":"30b189f7-8f15-4b3d-9fde-0d88a566ade4","Type":"ContainerDied","Data":"2d424d4fe8ccbdd8ffe8eaf340270903a87a1414f8beef66c4b0310bffd60ac5"} Oct 30 19:36:43.014030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:43.013996 4789 generic.go:356] "Generic (PLEG): container finished" podID="28b99ced-4252-4e9f-b941-73635f87cd13" containerID="e46b613ea16b3730ecbce602ec8f92f7d04e613950a173c4737682967fd81dcb" exitCode=0 Oct 30 19:36:43.014163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:43.014078 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" event={"ID":"28b99ced-4252-4e9f-b941-73635f87cd13","Type":"ContainerDied","Data":"e46b613ea16b3730ecbce602ec8f92f7d04e613950a173c4737682967fd81dcb"} Oct 30 19:36:44.206735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.206700 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:44.209973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.209949 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:44.338314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338232 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\") pod \"28b99ced-4252-4e9f-b941-73635f87cd13\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " Oct 30 19:36:44.338314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338295 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\" (UniqueName: \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\") pod \"28b99ced-4252-4e9f-b941-73635f87cd13\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " Oct 30 19:36:44.338564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338333 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\") pod \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " Oct 30 19:36:44.338564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338436 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\") pod \"28b99ced-4252-4e9f-b941-73635f87cd13\" (UID: \"28b99ced-4252-4e9f-b941-73635f87cd13\") " Oct 30 19:36:44.338564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338521 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\" (UniqueName: \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\") pod \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " Oct 30 19:36:44.338730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338563 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\") pod \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\" (UID: \"30b189f7-8f15-4b3d-9fde-0d88a566ade4\") " Oct 30 19:36:44.338955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.338919 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca" pod "28b99ced-4252-4e9f-b941-73635f87cd13" (UID: "28b99ced-4252-4e9f-b941-73635f87cd13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:44.339065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.339051 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca" pod "30b189f7-8f15-4b3d-9fde-0d88a566ade4" (UID: "30b189f7-8f15-4b3d-9fde-0d88a566ade4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:44.341290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.341259 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd" pod "28b99ced-4252-4e9f-b941-73635f87cd13" (UID: "28b99ced-4252-4e9f-b941-73635f87cd13"). InnerVolumeSpecName "kube-api-access-x4fcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:44.341396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.341369 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca" pod "30b189f7-8f15-4b3d-9fde-0d88a566ade4" (UID: "30b189f7-8f15-4b3d-9fde-0d88a566ade4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:44.341444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.341416 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca" pod "28b99ced-4252-4e9f-b941-73635f87cd13" (UID: "28b99ced-4252-4e9f-b941-73635f87cd13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:44.341626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.341604 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f" pod "30b189f7-8f15-4b3d-9fde-0d88a566ade4" (UID: "30b189f7-8f15-4b3d-9fde-0d88a566ade4"). InnerVolumeSpecName "kube-api-access-dmv5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:44.439613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439581 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:44.439613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439612 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmv5f\" (UniqueName: \"kubernetes.io/projected/30b189f7-8f15-4b3d-9fde-0d88a566ade4-kube-api-access-dmv5f\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:44.439805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439625 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:44.439805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439637 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b99ced-4252-4e9f-b941-73635f87cd13-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:44.439805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439646 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4fcd\" (UniqueName: \"kubernetes.io/projected/28b99ced-4252-4e9f-b941-73635f87cd13-kube-api-access-x4fcd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:44.439805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:44.439657 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b189f7-8f15-4b3d-9fde-0d88a566ade4-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:45.031347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.031313 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" Oct 30 19:36:45.031347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.031332 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799" event={"ID":"30b189f7-8f15-4b3d-9fde-0d88a566ade4","Type":"ContainerDied","Data":"049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2"} Oct 30 19:36:45.031644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.031383 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049ebc71ec224ee0792aa7a83f5d76ea5d3e0ef5df2390c8e0ef8984c940bcb2" Oct 30 19:36:45.032815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.032797 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" Oct 30 19:36:45.032924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.032835 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx" event={"ID":"28b99ced-4252-4e9f-b941-73635f87cd13","Type":"ContainerDied","Data":"391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5"} Oct 30 19:36:45.032924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:45.032878 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="391dea57594d44e3c8dc80766f99ea4a0611fee990756cbce176e3fbbc3533b5" Oct 30 19:36:48.196672 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:36:48.196630 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:36:50.700408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.700367 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5"] Oct 30 19:36:50.701080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701056 4789 cpu_manager.go:414] "containerMap: removing container" podUID="28b99ced-4252-4e9f-b941-73635f87cd13" containerName="revoke-keys" Oct 30 19:36:50.701179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701081 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b99ced-4252-4e9f-b941-73635f87cd13" containerName="revoke-keys" Oct 30 19:36:50.701179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701107 4789 cpu_manager.go:414] "containerMap: removing container" podUID="30b189f7-8f15-4b3d-9fde-0d88a566ade4" containerName="revoke-keys" Oct 30 19:36:50.701179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701114 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b189f7-8f15-4b3d-9fde-0d88a566ade4" containerName="revoke-keys" Oct 30 19:36:50.701413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701320 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b99ced-4252-4e9f-b941-73635f87cd13" containerName="revoke-keys" Oct 30 19:36:50.701413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.701340 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b189f7-8f15-4b3d-9fde-0d88a566ade4" containerName="revoke-keys" Oct 30 19:36:50.706488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.706467 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.758189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.758153 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5"] Oct 30 19:36:50.799422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.799384 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:50.799422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.799384 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:50.799659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.799425 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:50.802368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.802350 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:36:50.805922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.805887 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jxp\" (UniqueName: \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.806047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.806026 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.806142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.806121 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.907086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.907053 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.907086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.907091 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.907368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.907163 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jxp\" (UniqueName: \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.907766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.907741 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.909665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.909637 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:50.966052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:50.965966 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jxp\" (UniqueName: \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\") pod \"maas-api-revoke-sub-fd507597-mzkn5\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:51.020932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:51.020890 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:51.056249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:51.056199 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb"] Oct 30 19:36:51.061274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:51.061246 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602bbbac-qrqhb"] Oct 30 19:36:51.441383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:51.428680 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5"] Oct 30 19:36:51.672403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:51.672363 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72708a26-781a-4c4a-af03-7b623bef1678" path="/var/lib/kubelet/pods/72708a26-781a-4c4a-af03-7b623bef1678/volumes" Oct 30 19:36:52.085976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.085884 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" event={"ID":"aee77beb-32a5-40eb-88ff-1dec4f59147e","Type":"ContainerStarted","Data":"4f4b6ae5c40d6d0b3e9e9846a183b6b0794eca9da2cb004565e690a796748c4a"} Oct 30 19:36:52.085976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.085938 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" event={"ID":"aee77beb-32a5-40eb-88ff-1dec4f59147e","Type":"ContainerStarted","Data":"a4479e16277a98e45abdc789bb36f535bc124c3b8bf4c3915d2bd38a9206e06a"} Oct 30 19:36:52.197151 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.197080 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" podStartSLOduration=2.197062078 podStartE2EDuration="2.197062078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:52.195664705 +0000 UTC m=+3048.697575191" watchObservedRunningTime="2026-10-30 19:36:52.197062078 +0000 UTC m=+3048.698972565" Oct 30 19:36:52.314666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.314635 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98"] Oct 30 19:36:52.325512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.324527 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.383549 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.383462 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98"] Oct 30 19:36:52.430321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.430288 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.430492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.430446 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.430492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.430483 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbr5\" (UniqueName: \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.531845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.531811 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.532034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.531942 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.532034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.531970 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbr5\" (UniqueName: \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.532649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.532625 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.534346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.534323 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.590747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.590703 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbr5\" (UniqueName: \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\") pod \"maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:52.640357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:52.640260 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:53.019318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:53.019287 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98"] Oct 30 19:36:53.020174 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:36:53.020151 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb9dd1c_ab9c_4ecf_ab8d_e4553197a2d4.slice/crio-ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012 WatchSource:0}: Error finding container ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012: Status 404 returned error can't find the container with id ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012 Oct 30 19:36:53.093483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:53.093451 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" event={"ID":"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4","Type":"ContainerStarted","Data":"ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012"} Oct 30 19:36:53.095331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:53.095307 4789 generic.go:356] "Generic (PLEG): container finished" podID="aee77beb-32a5-40eb-88ff-1dec4f59147e" containerID="4f4b6ae5c40d6d0b3e9e9846a183b6b0794eca9da2cb004565e690a796748c4a" exitCode=0 Oct 30 19:36:53.095439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:53.095340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" event={"ID":"aee77beb-32a5-40eb-88ff-1dec4f59147e","Type":"ContainerDied","Data":"4f4b6ae5c40d6d0b3e9e9846a183b6b0794eca9da2cb004565e690a796748c4a"} Oct 30 19:36:54.114283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.113307 4789 generic.go:356] "Generic (PLEG): container finished" podID="2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" containerID="bacd35b0d1becec43338aa5aadda4445b3c3a4587f1d6734083b0e85b8b1c402" exitCode=0 Oct 30 19:36:54.114283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.113394 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" event={"ID":"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4","Type":"ContainerDied","Data":"bacd35b0d1becec43338aa5aadda4445b3c3a4587f1d6734083b0e85b8b1c402"} Oct 30 19:36:54.297346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.297314 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:54.459456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.459425 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\" (UniqueName: \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\") pod \"aee77beb-32a5-40eb-88ff-1dec4f59147e\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " Oct 30 19:36:54.459648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.459481 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\") pod \"aee77beb-32a5-40eb-88ff-1dec4f59147e\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " Oct 30 19:36:54.459648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.459532 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\") pod \"aee77beb-32a5-40eb-88ff-1dec4f59147e\" (UID: \"aee77beb-32a5-40eb-88ff-1dec4f59147e\") " Oct 30 19:36:54.459949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.459924 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca" pod "aee77beb-32a5-40eb-88ff-1dec4f59147e" (UID: "aee77beb-32a5-40eb-88ff-1dec4f59147e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:54.461760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.461731 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca" pod "aee77beb-32a5-40eb-88ff-1dec4f59147e" (UID: "aee77beb-32a5-40eb-88ff-1dec4f59147e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:54.461875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.461756 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp" pod "aee77beb-32a5-40eb-88ff-1dec4f59147e" (UID: "aee77beb-32a5-40eb-88ff-1dec4f59147e"). InnerVolumeSpecName "kube-api-access-p2jxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:54.560829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.560796 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jxp\" (UniqueName: \"kubernetes.io/projected/aee77beb-32a5-40eb-88ff-1dec4f59147e-kube-api-access-p2jxp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:54.560829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.560821 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:54.560829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:54.560832 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aee77beb-32a5-40eb-88ff-1dec4f59147e-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:55.122623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.122588 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" event={"ID":"aee77beb-32a5-40eb-88ff-1dec4f59147e","Type":"ContainerDied","Data":"a4479e16277a98e45abdc789bb36f535bc124c3b8bf4c3915d2bd38a9206e06a"} Oct 30 19:36:55.122623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.122623 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5" Oct 30 19:36:55.123123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.122650 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4479e16277a98e45abdc789bb36f535bc124c3b8bf4c3915d2bd38a9206e06a" Oct 30 19:36:55.267930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.267899 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:55.369228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.369187 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\" (UniqueName: \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\") pod \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " Oct 30 19:36:55.369434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.369255 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\") pod \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " Oct 30 19:36:55.369434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.369360 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\") pod \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\" (UID: \"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4\") " Oct 30 19:36:55.369593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.369573 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca" pod "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" (UID: "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:55.369871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.369846 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:55.371372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.371350 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca" pod "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" (UID: "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:55.371445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.371418 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5" pod "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" (UID: "2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4"). InnerVolumeSpecName "kube-api-access-brbr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:55.471483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.471443 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brbr5\" (UniqueName: \"kubernetes.io/projected/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-kube-api-access-brbr5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:55.471483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:55.471482 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:56.132368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:56.132337 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" Oct 30 19:36:56.132368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:56.132361 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98" event={"ID":"2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4","Type":"ContainerDied","Data":"ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012"} Oct 30 19:36:56.132927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:36:56.132418 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0d27e0994bc7846b674097330e142f9a0b89dd9788fcaa0c9a07750aae5012" Oct 30 19:37:01.494537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.494506 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v"] Oct 30 19:37:01.495013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.494985 4789 cpu_manager.go:414] "containerMap: removing container" podUID="aee77beb-32a5-40eb-88ff-1dec4f59147e" containerName="revoke-keys" Oct 30 19:37:01.495013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.494999 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee77beb-32a5-40eb-88ff-1dec4f59147e" containerName="revoke-keys" Oct 30 19:37:01.495090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.495028 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" containerName="revoke-keys" Oct 30 19:37:01.495090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.495033 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" containerName="revoke-keys" Oct 30 19:37:01.495159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.495120 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee77beb-32a5-40eb-88ff-1dec4f59147e" containerName="revoke-keys" Oct 30 19:37:01.495159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.495131 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" containerName="revoke-keys" Oct 30 19:37:01.499808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.499788 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.596138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.596110 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:01.596412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.596126 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:01.596412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.596148 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:01.596598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.596299 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:37:01.608567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.608538 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v"] Oct 30 19:37:01.639786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.639733 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.640240 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.640147 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.640553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.640514 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bvq\" (UniqueName: \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.741906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.741867 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.742092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.741926 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.742092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.741969 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bvq\" (UniqueName: \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.742728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.742691 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.744329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.744303 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.800405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.800338 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bvq\" (UniqueName: \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\") pod \"maas-api-revoke-sub-c8f264a3-4bb8v\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.815522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.815493 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:01.944381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.944339 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:37:01.949061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.949033 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:01.999454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.999406 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-db06cf-dockercfg-bz4vq\"" Oct 30 19:37:01.999454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:01.999425 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-db06cf\"" Oct 30 19:37:02.001894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.001868 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:37:02.055865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.055774 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.056253 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.056191 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8gz\" (UniqueName: \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.157815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.157778 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8gz\" (UniqueName: \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.158082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.157880 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.158715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.158687 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.198903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.198875 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v"] Oct 30 19:37:02.199866 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:02.199837 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8d456b_8657_4576_97de_a57bbc104293.slice/crio-10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0 WatchSource:0}: Error finding container 10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0: Status 404 returned error can't find the container with id 10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0 Oct 30 19:37:02.216438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.216413 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8gz\" (UniqueName: \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\") pod \"payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.266244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.266193 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:02.638305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:02.638276 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:37:02.639522 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:02.639495 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d1d59f_1e56_41c0_ade3_10fa0ffe5e62.slice/crio-7d9f9cc9e6350cac9781c47a756874a318534ccbd25bb19291d89235491ab173 WatchSource:0}: Error finding container 7d9f9cc9e6350cac9781c47a756874a318534ccbd25bb19291d89235491ab173: Status 404 returned error can't find the container with id 7d9f9cc9e6350cac9781c47a756874a318534ccbd25bb19291d89235491ab173 Oct 30 19:37:03.188394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.188355 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" event={"ID":"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62","Type":"ContainerStarted","Data":"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf"} Oct 30 19:37:03.188567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.188429 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:03.188567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.188457 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" event={"ID":"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62","Type":"ContainerStarted","Data":"7d9f9cc9e6350cac9781c47a756874a318534ccbd25bb19291d89235491ab173"} Oct 30 19:37:03.190087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.190056 4789 generic.go:356] "Generic (PLEG): container finished" podID="5a8d456b-8657-4576-97de-a57bbc104293" containerID="3405e419b15f6d774d98fe1c3a6768db6234e9346d42b8c8d372908a2ac7115a" exitCode=0 Oct 30 19:37:03.190231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.190133 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" event={"ID":"5a8d456b-8657-4576-97de-a57bbc104293","Type":"ContainerDied","Data":"3405e419b15f6d774d98fe1c3a6768db6234e9346d42b8c8d372908a2ac7115a"} Oct 30 19:37:03.190231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.190182 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" event={"ID":"5a8d456b-8657-4576-97de-a57bbc104293","Type":"ContainerStarted","Data":"10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0"} Oct 30 19:37:03.303452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:03.303402 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" podStartSLOduration=2.303388485 podStartE2EDuration="2.303388485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:03.299998534 +0000 UTC m=+3059.801909032" watchObservedRunningTime="2026-10-30 19:37:03.303388485 +0000 UTC m=+3059.805299013" Oct 30 19:37:04.368337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.368314 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:04.381369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.381347 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\") pod \"5a8d456b-8657-4576-97de-a57bbc104293\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " Oct 30 19:37:04.381496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.381398 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\" (UniqueName: \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\") pod \"5a8d456b-8657-4576-97de-a57bbc104293\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " Oct 30 19:37:04.381564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.381524 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\") pod \"5a8d456b-8657-4576-97de-a57bbc104293\" (UID: \"5a8d456b-8657-4576-97de-a57bbc104293\") " Oct 30 19:37:04.381738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.381716 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca" pod "5a8d456b-8657-4576-97de-a57bbc104293" (UID: "5a8d456b-8657-4576-97de-a57bbc104293"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:04.381997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.381980 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:04.383397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.383374 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq" pod "5a8d456b-8657-4576-97de-a57bbc104293" (UID: "5a8d456b-8657-4576-97de-a57bbc104293"). InnerVolumeSpecName "kube-api-access-62bvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:04.383587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.383565 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca" pod "5a8d456b-8657-4576-97de-a57bbc104293" (UID: "5a8d456b-8657-4576-97de-a57bbc104293"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:04.482990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.482917 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bvq\" (UniqueName: \"kubernetes.io/projected/5a8d456b-8657-4576-97de-a57bbc104293-kube-api-access-62bvq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:04.482990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:04.482955 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d456b-8657-4576-97de-a57bbc104293-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:05.205553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:05.205525 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" Oct 30 19:37:05.205733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:05.205564 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v" event={"ID":"5a8d456b-8657-4576-97de-a57bbc104293","Type":"ContainerDied","Data":"10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0"} Oct 30 19:37:05.205733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:05.205636 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10def942a326e3218e05f83cc9ddb1ad0cd4744828bd51dbb17ea0cd02cf5bf0" Oct 30 19:37:06.394053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.394023 4789 scope.go:122] "RemoveContainer" containerID="34fd03566ac7e15b6f3d458df634788d2a5721e4fd73e43ed47e56d17d3a9183" Oct 30 19:37:06.921728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.921698 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94"] Oct 30 19:37:06.922165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.922151 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5a8d456b-8657-4576-97de-a57bbc104293" containerName="revoke-keys" Oct 30 19:37:06.922234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.922167 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8d456b-8657-4576-97de-a57bbc104293" containerName="revoke-keys" Oct 30 19:37:06.922301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.922291 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8d456b-8657-4576-97de-a57bbc104293" containerName="revoke-keys" Oct 30 19:37:06.927026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:06.927001 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.006851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.006817 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.007023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.006877 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4c5\" (UniqueName: \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.007023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.007009 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.023785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.023757 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:37:07.023785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.023775 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:07.023785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.023775 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:07.024013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.023775 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:07.027457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.027439 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94"] Oct 30 19:37:07.108428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.108400 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.108594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.108450 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4c5\" (UniqueName: \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.108594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.108497 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.109165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.109133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.110920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.110898 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.166652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.166622 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4c5\" (UniqueName: \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\") pod \"maas-api-revoke-sub-341fa872-k2m94\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.241414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.241389 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:07.616489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:07.616461 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94"] Oct 30 19:37:07.617390 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:07.617359 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc662fb6e_1fa0_4c7c_9024_a7d072b870c9.slice/crio-3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72 WatchSource:0}: Error finding container 3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72: Status 404 returned error can't find the container with id 3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72 Oct 30 19:37:08.231189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:08.231155 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" event={"ID":"c662fb6e-1fa0-4c7c-9024-a7d072b870c9","Type":"ContainerStarted","Data":"196e0438cf15631c7ce59cc42e8d92f9ab4cc9b98215b9c31df21afc6d4bb0fb"} Oct 30 19:37:08.231390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:08.231221 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" event={"ID":"c662fb6e-1fa0-4c7c-9024-a7d072b870c9","Type":"ContainerStarted","Data":"3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72"} Oct 30 19:37:08.343618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:08.343563 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" podStartSLOduration=2.343548621 podStartE2EDuration="2.343548621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:08.34039999 +0000 UTC m=+3064.842310476" watchObservedRunningTime="2026-10-30 19:37:08.343548621 +0000 UTC m=+3064.845459107" Oct 30 19:37:09.248520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:09.248488 4789 generic.go:356] "Generic (PLEG): container finished" podID="c662fb6e-1fa0-4c7c-9024-a7d072b870c9" containerID="196e0438cf15631c7ce59cc42e8d92f9ab4cc9b98215b9c31df21afc6d4bb0fb" exitCode=0 Oct 30 19:37:09.248921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:09.248577 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" event={"ID":"c662fb6e-1fa0-4c7c-9024-a7d072b870c9","Type":"ContainerDied","Data":"196e0438cf15631c7ce59cc42e8d92f9ab4cc9b98215b9c31df21afc6d4bb0fb"} Oct 30 19:37:10.416626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.416599 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:10.443673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.443643 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\") pod \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " Oct 30 19:37:10.443871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.443739 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\" (UniqueName: \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\") pod \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " Oct 30 19:37:10.443871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.443828 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\") pod \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\" (UID: \"c662fb6e-1fa0-4c7c-9024-a7d072b870c9\") " Oct 30 19:37:10.444079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.444057 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca" pod "c662fb6e-1fa0-4c7c-9024-a7d072b870c9" (UID: "c662fb6e-1fa0-4c7c-9024-a7d072b870c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:10.444302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.444282 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:10.445855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.445832 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca" pod "c662fb6e-1fa0-4c7c-9024-a7d072b870c9" (UID: "c662fb6e-1fa0-4c7c-9024-a7d072b870c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:10.445940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.445873 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5" pod "c662fb6e-1fa0-4c7c-9024-a7d072b870c9" (UID: "c662fb6e-1fa0-4c7c-9024-a7d072b870c9"). InnerVolumeSpecName "kube-api-access-pc4c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:10.545545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.545465 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:10.545545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:10.545492 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4c5\" (UniqueName: \"kubernetes.io/projected/c662fb6e-1fa0-4c7c-9024-a7d072b870c9-kube-api-access-pc4c5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:11.264295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:11.264262 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" Oct 30 19:37:11.264295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:11.264284 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94" event={"ID":"c662fb6e-1fa0-4c7c-9024-a7d072b870c9","Type":"ContainerDied","Data":"3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72"} Oct 30 19:37:11.264510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:11.264337 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3116446b6630d43fba295a5ed9e81e53cfedd3957ed8dd16aa1cb25d79d79a72" Oct 30 19:37:12.613313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.613283 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:37:12.613801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.613788 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c662fb6e-1fa0-4c7c-9024-a7d072b870c9" containerName="revoke-keys" Oct 30 19:37:12.613847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.613802 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c662fb6e-1fa0-4c7c-9024-a7d072b870c9" containerName="revoke-keys" Oct 30 19:37:12.613890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.613881 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c662fb6e-1fa0-4c7c-9024-a7d072b870c9" containerName="revoke-keys" Oct 30 19:37:12.617481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.617460 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.667845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.667817 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.667964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.667850 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.667964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.667940 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668078 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668118 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668221 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668262 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668291 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcg4\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.668394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.668355 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.712310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.712280 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:37:12.769080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769048 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769086 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769184 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769322 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769351 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769398 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769444 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769474 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcg4\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769506 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769591 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769611 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.769882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.769811 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.770047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.770023 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.770119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.770025 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.771769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.771744 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.771871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.771800 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.827828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.827792 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.828009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.827932 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcg4\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\") pod \"e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:12.931245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:12.931120 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:13.360519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:13.360491 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:37:13.360931 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:13.360909 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb63e789_972b_49a4_a71e_a96b78304dcb.slice/crio-acd7b0b866be5b5423c2933aa9129a51123807e270091201d48a27eaf8165072 WatchSource:0}: Error finding container acd7b0b866be5b5423c2933aa9129a51123807e270091201d48a27eaf8165072: Status 404 returned error can't find the container with id acd7b0b866be5b5423c2933aa9129a51123807e270091201d48a27eaf8165072 Oct 30 19:37:13.363392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:13.363359 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:13.363520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:13.363436 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:13.363520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:13.363488 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:14.239232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.221410 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:37:14.288588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.288544 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" event={"ID":"bb63e789-972b-49a4-a71e-a96b78304dcb","Type":"ContainerStarted","Data":"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd"} Oct 30 19:37:14.288875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.288856 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" event={"ID":"bb63e789-972b-49a4-a71e-a96b78304dcb","Type":"ContainerStarted","Data":"acd7b0b866be5b5423c2933aa9129a51123807e270091201d48a27eaf8165072"} Oct 30 19:37:14.459248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.459126 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podStartSLOduration=2.459108977 podStartE2EDuration="2.459108977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:14.456804116 +0000 UTC m=+3070.958714602" watchObservedRunningTime="2026-10-30 19:37:14.459108977 +0000 UTC m=+3070.961019465" Oct 30 19:37:14.933385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.933352 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:14.933738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.933715 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:14.933892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:14.933867 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:15.932319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:15.932280 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:15.932817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:15.932342 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:16.932232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:16.932174 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:16.932407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:16.932271 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:17.931798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:17.931756 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:17.932064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:17.931823 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:18.224080 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:37:18.224041 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:37:18.935234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:18.931645 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:18.935234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:18.931727 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:19.932174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:19.932123 4789 patch_prober.go:29] interesting pod/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 19:37:19.932720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:19.932202 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 19:37:21.870811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:21.870764 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:21.871299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:21.871111 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:21.872414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:21.872391 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:37:23.354422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.354391 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:37:23.354836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.354581 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" podUID="f9a9041c-2649-417c-97a6-c4acd578bc3a" containerName="payload-processing" containerID="cri-o://ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796" gracePeriod=30 Oct 30 19:37:23.626640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.626615 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:37:23.766082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.766050 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\") pod \"f9a9041c-2649-417c-97a6-c4acd578bc3a\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " Oct 30 19:37:23.766236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.766137 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\" (UniqueName: \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\") pod \"f9a9041c-2649-417c-97a6-c4acd578bc3a\" (UID: \"f9a9041c-2649-417c-97a6-c4acd578bc3a\") " Oct 30 19:37:23.766491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.766462 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume" pod "f9a9041c-2649-417c-97a6-c4acd578bc3a" (UID: "f9a9041c-2649-417c-97a6-c4acd578bc3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:23.766605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.766588 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a9041c-2649-417c-97a6-c4acd578bc3a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:23.768344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.768320 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s" pod "f9a9041c-2649-417c-97a6-c4acd578bc3a" (UID: "f9a9041c-2649-417c-97a6-c4acd578bc3a"). InnerVolumeSpecName "kube-api-access-4tz8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:23.867729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:23.867629 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tz8s\" (UniqueName: \"kubernetes.io/projected/f9a9041c-2649-417c-97a6-c4acd578bc3a-kube-api-access-4tz8s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:24.390337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.390301 4789 generic.go:356] "Generic (PLEG): container finished" podID="f9a9041c-2649-417c-97a6-c4acd578bc3a" containerID="ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796" exitCode=0 Oct 30 19:37:24.390337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.390341 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" event={"ID":"f9a9041c-2649-417c-97a6-c4acd578bc3a","Type":"ContainerDied","Data":"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796"} Oct 30 19:37:24.390971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.390362 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" Oct 30 19:37:24.390971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.390402 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7" event={"ID":"f9a9041c-2649-417c-97a6-c4acd578bc3a","Type":"ContainerDied","Data":"0f80ff7d4a13080b87f8675458bfb4f7c8963cf7765cbc3db29b6f1f56ca4ec9"} Oct 30 19:37:24.390971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.390422 4789 scope.go:122] "RemoveContainer" containerID="ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796" Oct 30 19:37:24.401290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.401274 4789 scope.go:122] "RemoveContainer" containerID="ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796" Oct 30 19:37:24.401575 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:37:24.401553 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796\": container with ID starting with ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796 not found: ID does not exist" containerID="ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796" Oct 30 19:37:24.401637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.401586 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796"} err="failed to get container status \"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796\": rpc error: code = NotFound desc = could not find container \"ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796\": container with ID starting with ce07ecd1cd7a71bab80949ce2fbc9ab1dea8e6d4b1fffefdce9b4d08af8fe796 not found: ID does not exist" Oct 30 19:37:24.547382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.547355 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:37:24.593484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:24.593460 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-83ac32-655bc644d8-2kjt7"] Oct 30 19:37:25.306266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.306230 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:37:25.306731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.306715 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f9a9041c-2649-417c-97a6-c4acd578bc3a" containerName="payload-processing" Oct 30 19:37:25.306731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.306730 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a9041c-2649-417c-97a6-c4acd578bc3a" containerName="payload-processing" Oct 30 19:37:25.306894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.306820 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a9041c-2649-417c-97a6-c4acd578bc3a" containerName="payload-processing" Oct 30 19:37:25.311527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.311505 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382094 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgvc\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382185 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382226 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382273 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382329 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382502 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382643 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382688 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.382867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.382845 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.398034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.398008 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fe7634-openshift-default-dockercfg-dkbxj\"" Oct 30 19:37:25.412063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.412040 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:37:25.484368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484339 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgvc\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484376 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484400 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484454 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484481 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484545 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484638 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484674 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.484779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484760 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.485180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.484801 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.485180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.485074 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.485180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.485133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.485347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.485205 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.485347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.485304 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.487676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.487652 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.487798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.487657 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.557282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.555288 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.557282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.556929 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgvc\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\") pod \"e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.625445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.625413 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:25.672392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:25.672360 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a9041c-2649-417c-97a6-c4acd578bc3a" path="/var/lib/kubelet/pods/f9a9041c-2649-417c-97a6-c4acd578bc3a/volumes" Oct 30 19:37:26.064233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.063944 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98"] Oct 30 19:37:26.069842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.069812 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-83ac32-zdn98"] Oct 30 19:37:26.074341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.074318 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:37:26.091024 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:26.090976 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c57b60a_be79_46a7_999b_639942a1c038.slice/crio-3bc242d52077b67f8b865f1f72fc4535c013bff02219704111b7a6d807a2cfdf WatchSource:0}: Error finding container 3bc242d52077b67f8b865f1f72fc4535c013bff02219704111b7a6d807a2cfdf: Status 404 returned error can't find the container with id 3bc242d52077b67f8b865f1f72fc4535c013bff02219704111b7a6d807a2cfdf Oct 30 19:37:26.094286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.094253 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:26.094415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.094341 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:26.094415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.094374 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:26.377692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.377619 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:37:26.382613 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.382587 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.410864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.410833 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerStarted","Data":"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44"} Oct 30 19:37:26.411298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.410889 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerStarted","Data":"3bc242d52077b67f8b865f1f72fc4535c013bff02219704111b7a6d807a2cfdf"} Oct 30 19:37:26.439137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.439106 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-2186d8\"" Oct 30 19:37:26.440812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.440748 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-2186d8-dockercfg-gtdj6\"" Oct 30 19:37:26.466263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.462763 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:37:26.497907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.497870 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.498153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.497921 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.598853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.598820 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.599062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.598864 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.599682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.599650 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.625934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.625912 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:26.640107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.640009 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podStartSLOduration=1.63999396 podStartE2EDuration="1.63999396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:26.639399672 +0000 UTC m=+3083.141310157" watchObservedRunningTime="2026-10-30 19:37:26.63999396 +0000 UTC m=+3083.141904641" Oct 30 19:37:26.656295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.656269 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\") pod \"payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:26.696397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:26.696368 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:27.051803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.051761 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc"] Oct 30 19:37:27.055160 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.055137 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcc70a0-k4qfc"] Oct 30 19:37:27.072390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.070390 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:37:27.187004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.186978 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:37:27.190877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.190862 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.239964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.239944 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9275c3-openshift-default-dockercfg-hq6vr\"" Oct 30 19:37:27.256958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.256931 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:37:27.306551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306493 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306542 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306561 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306661 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306700 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbz9\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306813 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306839 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.306870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306864 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.307006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.306956 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407559 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407600 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407649 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407688 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbz9\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407760 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407778 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407800 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.407952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.407858 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.408180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.408026 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.408180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.408144 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.408416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.408390 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.408492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.408417 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.408553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.408516 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.410052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.410031 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.410186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.410168 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.419015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.418988 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" event={"ID":"a59688af-426d-4856-b0a2-8fd334be746a","Type":"ContainerStarted","Data":"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c"} Oct 30 19:37:27.419361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.419037 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" event={"ID":"a59688af-426d-4856-b0a2-8fd334be746a","Type":"ContainerStarted","Data":"39045af7242a77ad6f6be6d451972f698a51fee51a125cd9afa97348ea044c6c"} Oct 30 19:37:27.419361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.419085 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:27.465448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.465362 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.465543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.465528 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbz9\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\") pod \"e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.505231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.505189 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:27.631664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.631640 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:27.670181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.670140 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4" path="/var/lib/kubelet/pods/2cb9dd1c-ab9c-4ecf-ab8d-e4553197a2d4/volumes" Oct 30 19:37:27.670826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.670798 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92672c4e-204a-4e2e-a9ad-51f08dff8f80" path="/var/lib/kubelet/pods/92672c4e-204a-4e2e-a9ad-51f08dff8f80/volumes" Oct 30 19:37:27.753047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.752998 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" podStartSLOduration=1.752984181 podStartE2EDuration="1.752984181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:27.554100134 +0000 UTC m=+3084.056010621" watchObservedRunningTime="2026-10-30 19:37:27.752984181 +0000 UTC m=+3084.254894668" Oct 30 19:37:27.758459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.758435 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:37:27.764069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.764051 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.811943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.811911 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.811960 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812063 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812111 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812264 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812154 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cbb\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812262 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812288 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812344 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.812428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.812381 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.818414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.818392 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:37:27.899839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.899817 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:37:27.901794 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:27.901759 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1812787_bdf2_4b94_84d0_095bfe4ba703.slice/crio-c28305814f7bbf67cad1c10aa6befe0084213f64f2e89f9076acbb3ff4f9112c WatchSource:0}: Error finding container c28305814f7bbf67cad1c10aa6befe0084213f64f2e89f9076acbb3ff4f9112c: Status 404 returned error can't find the container with id c28305814f7bbf67cad1c10aa6befe0084213f64f2e89f9076acbb3ff4f9112c Oct 30 19:37:27.905047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.905010 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:27.905170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.905092 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:27.905170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.905138 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:27.913625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913605 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.913733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913639 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.913733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913664 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.913733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913709 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.913901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913833 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.913901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913870 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913900 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913931 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913967 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.913968 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cbb\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.914070 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.914369 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.914567 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.914696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.914610 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.916723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.916694 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.916844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.916827 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.971689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.971661 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:27.971931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:27.971913 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cbb\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\") pod \"e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:28.078479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.078448 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:28.429063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.428968 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" event={"ID":"d1812787-bdf2-4b94-84d0-095bfe4ba703","Type":"ContainerStarted","Data":"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62"} Oct 30 19:37:28.429063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.429029 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" event={"ID":"d1812787-bdf2-4b94-84d0-095bfe4ba703","Type":"ContainerStarted","Data":"c28305814f7bbf67cad1c10aa6befe0084213f64f2e89f9076acbb3ff4f9112c"} Oct 30 19:37:28.429516 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.429345 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:28.430424 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.430407 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:37:28.469337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.469316 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:37:28.469998 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:37:28.469974 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb62e1c2_f90c_498f_acda_6c0e655bf29f.slice/crio-cf613994ac6a6371b93e5b8b2a15ef948ca4926cd084dc6157fe75734ab52e71 WatchSource:0}: Error finding container cf613994ac6a6371b93e5b8b2a15ef948ca4926cd084dc6157fe75734ab52e71: Status 404 returned error can't find the container with id cf613994ac6a6371b93e5b8b2a15ef948ca4926cd084dc6157fe75734ab52e71 Oct 30 19:37:28.472268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.472238 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:28.472358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.472307 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:28.472358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.472336 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:37:28.514711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.514674 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:28.518452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.518321 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:28.518452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.518378 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:28.546373 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:28.546322 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podStartSLOduration=1.546306358 podStartE2EDuration="1.546306358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:28.542615742 +0000 UTC m=+3085.044526253" watchObservedRunningTime="2026-10-30 19:37:28.546306358 +0000 UTC m=+3085.048216848" Oct 30 19:37:29.458255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:29.458191 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" event={"ID":"fb62e1c2-f90c-498f-acda-6c0e655bf29f","Type":"ContainerStarted","Data":"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e"} Oct 30 19:37:29.458781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:29.458286 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" event={"ID":"fb62e1c2-f90c-498f-acda-6c0e655bf29f","Type":"ContainerStarted","Data":"cf613994ac6a6371b93e5b8b2a15ef948ca4926cd084dc6157fe75734ab52e71"} Oct 30 19:37:29.512061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:29.511625 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:29.512061 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:29.511686 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:29.626109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:29.626017 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podStartSLOduration=2.625993034 podStartE2EDuration="2.625993034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:29.577719311 +0000 UTC m=+3086.079629794" watchObservedRunningTime="2026-10-30 19:37:29.625993034 +0000 UTC m=+3086.127903522" Oct 30 19:37:30.079236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:30.079190 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:30.080491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:30.080463 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:30.080621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:30.080521 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:30.553399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:30.553323 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:30.553399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:30.553385 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:31.079576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:31.079538 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:31.079760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:31.079594 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:31.511233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:31.510588 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:31.511233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:31.510650 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:32.079259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:32.079018 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:32.079259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:32.079085 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:32.505554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:32.505520 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:32.505861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:32.505829 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:33.078698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:33.078663 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:33.078885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:33.078720 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:33.509232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:33.507299 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:33.509232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:33.507358 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:34.079759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:34.079723 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.080092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:34.080056 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:34.505634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:34.505512 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.505634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:34.505572 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 19:37:35.078749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:35.078713 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:37:35.079126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:35.078788 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:37:35.522921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:35.522887 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:35.525146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:35.525049 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:35.539239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:35.534476 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:37:36.082794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:36.082765 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:36.083323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:36.083148 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:36.083805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:36.083786 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:37:38.432692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:38.432661 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:37:46.052697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:46.052624 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8"] Oct 30 19:37:46.080777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:46.080751 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ff307ec-mjwx8"] Oct 30 19:37:47.668597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:37:47.668569 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e2dd7f-73cc-498f-9136-86a16f8569d8" path="/var/lib/kubelet/pods/82e2dd7f-73cc-498f-9136-86a16f8569d8/volumes" Oct 30 19:37:48.195884 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:37:48.195843 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:38:04.602905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.602863 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:38:04.607718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.607699 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.660031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.660005 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:38:04.702457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.702408 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-af72cf\"" Oct 30 19:38:04.702639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.702575 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-af72cf-dockercfg-kf8n7\"" Oct 30 19:38:04.746044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.746002 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72n4\" (UniqueName: \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.746279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.746174 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.847884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.847849 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72n4\" (UniqueName: \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.848096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.847928 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.848647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.848623 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.906544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.906480 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72n4\" (UniqueName: \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\") pod \"payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:04.922300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:04.922274 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:05.311302 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:38:05.311270 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c017e7_55aa_49ed_bb53_3f12516a8d67.slice/crio-b948ff394b1c4549b8630a2227ab061b1731b8ae33e06a8b9f24c68859ad631d WatchSource:0}: Error finding container b948ff394b1c4549b8630a2227ab061b1731b8ae33e06a8b9f24c68859ad631d: Status 404 returned error can't find the container with id b948ff394b1c4549b8630a2227ab061b1731b8ae33e06a8b9f24c68859ad631d Oct 30 19:38:05.312311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:05.312281 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:38:05.763605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:05.763571 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" event={"ID":"d7c017e7-55aa-49ed-bb53-3f12516a8d67","Type":"ContainerStarted","Data":"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225"} Oct 30 19:38:05.764000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:05.763628 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" event={"ID":"d7c017e7-55aa-49ed-bb53-3f12516a8d67","Type":"ContainerStarted","Data":"b948ff394b1c4549b8630a2227ab061b1731b8ae33e06a8b9f24c68859ad631d"} Oct 30 19:38:05.764000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:05.763656 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:05.876515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:05.876459 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" podStartSLOduration=1.876444693 podStartE2EDuration="1.876444693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:05.874158449 +0000 UTC m=+3122.376068934" watchObservedRunningTime="2026-10-30 19:38:05.876444693 +0000 UTC m=+3122.378355208" Oct 30 19:38:06.430706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:06.430675 4789 scope.go:122] "RemoveContainer" containerID="72955da42693e7df070569c5d9a335b8b900d43319779efae00a1d5b2bc0234b" Oct 30 19:38:06.440505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:06.440479 4789 scope.go:122] "RemoveContainer" containerID="2b2a0a5f23bfbd409c059f65767fe4d82eef5df5daf1b6eee1de8f50c392db79" Oct 30 19:38:06.451989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:06.451968 4789 scope.go:122] "RemoveContainer" containerID="81b3940187a0e4f5d3f242b3646a09b6be4f592b2c58d85cccd88743cf2ff2f1" Oct 30 19:38:12.052342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:12.052310 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz"] Oct 30 19:38:12.098523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:12.098493 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-290921f6-wxztz"] Oct 30 19:38:13.669522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:13.669489 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d470778a-ab02-4b6d-a865-51eb7c27aadf" path="/var/lib/kubelet/pods/d470778a-ab02-4b6d-a865-51eb7c27aadf/volumes" Oct 30 19:38:16.773326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:16.773288 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:38:18.200917 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:38:18.200868 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:38:25.935991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:25.935952 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:38:25.943200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:25.943174 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:25.992452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:25.992419 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b2604c-openshift-default-dockercfg-cn8tl\"" Oct 30 19:38:26.006163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006130 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vm5\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006185 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006316 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006384 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006436 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006505 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006683 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006723 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.006768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.006757 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.045974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.045937 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:38:26.170665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.170626 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173358 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173413 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173478 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vm5\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173532 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173573 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173629 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173657 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.174505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.173689 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.175501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.175474 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.175907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.175888 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.177405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.177383 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.185939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.178021 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.186174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.185629 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.186297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.185793 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.201478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.174245 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.235492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.235440 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vm5\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.246237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.246180 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\") pod \"e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.260897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.260862 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:26.743959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.743921 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:38:26.757328 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:38:26.757298 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5a5f33_6e4c_4238_823e_feb61db1cdd3.slice/crio-fd68cf0a76aad0425938b0dbb50e25c7c2aa75fb22a2d1da6a34036ac224a3e2 WatchSource:0}: Error finding container fd68cf0a76aad0425938b0dbb50e25c7c2aa75fb22a2d1da6a34036ac224a3e2: Status 404 returned error can't find the container with id fd68cf0a76aad0425938b0dbb50e25c7c2aa75fb22a2d1da6a34036ac224a3e2 Oct 30 19:38:26.759916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.759883 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:26.760017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.759952 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:26.760017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.759980 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:38:26.947816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.947778 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerStarted","Data":"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907"} Oct 30 19:38:26.948228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:26.947841 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerStarted","Data":"fd68cf0a76aad0425938b0dbb50e25c7c2aa75fb22a2d1da6a34036ac224a3e2"} Oct 30 19:38:27.087853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:27.087734 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podStartSLOduration=2.087720059 podStartE2EDuration="2.087720059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:27.085607926 +0000 UTC m=+3143.587518411" watchObservedRunningTime="2026-10-30 19:38:27.087720059 +0000 UTC m=+3143.589630545" Oct 30 19:38:27.261478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:27.261411 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:27.263567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:27.263538 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:38:27.263715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:27.263584 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:38:28.056429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:28.056399 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9"] Oct 30 19:38:28.104530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:28.104499 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a076be9c-8fdc9"] Oct 30 19:38:28.265892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:28.265864 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:28.962201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:28.962162 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:28.963119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:28.963101 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:38:29.668581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:29.668549 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5127e126-a0c4-41df-bc8e-9d86c0e581ab" path="/var/lib/kubelet/pods/5127e126-a0c4-41df-bc8e-9d86c0e581ab/volumes" Oct 30 19:38:43.857591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.857559 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:38:43.861653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.861634 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:43.892510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.892481 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:43.892655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.892619 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9c8r\" (UniqueName: \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:43.918094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.918057 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:38:43.958219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.958172 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-b2604c-dockercfg-97tj7\"" Oct 30 19:38:43.958219 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.958173 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b2604c\"" Oct 30 19:38:43.993769 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.993740 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:43.993965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.993802 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9c8r\" (UniqueName: \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:43.994529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:43.994502 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:44.052599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:44.052561 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9c8r\" (UniqueName: \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\") pod \"payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:44.175706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:44.175628 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:44.557289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:44.557264 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:38:44.558318 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:38:44.558291 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd3ef15_5711_43f4_9141_8b266ed683fb.slice/crio-05486ef20cf2eb4d4b88d9781762c7b0ba053938b9acff36f47fb8c8a0c41313 WatchSource:0}: Error finding container 05486ef20cf2eb4d4b88d9781762c7b0ba053938b9acff36f47fb8c8a0c41313: Status 404 returned error can't find the container with id 05486ef20cf2eb4d4b88d9781762c7b0ba053938b9acff36f47fb8c8a0c41313 Oct 30 19:38:45.078006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:45.077964 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" event={"ID":"dbd3ef15-5711-43f4-9141-8b266ed683fb","Type":"ContainerStarted","Data":"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148"} Oct 30 19:38:45.078538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:45.078020 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" event={"ID":"dbd3ef15-5711-43f4-9141-8b266ed683fb","Type":"ContainerStarted","Data":"05486ef20cf2eb4d4b88d9781762c7b0ba053938b9acff36f47fb8c8a0c41313"} Oct 30 19:38:45.078538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:45.078074 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:38:45.191525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:45.191468 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" podStartSLOduration=2.191451767 podStartE2EDuration="2.191451767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:45.187868649 +0000 UTC m=+3161.689779129" watchObservedRunningTime="2026-10-30 19:38:45.191451767 +0000 UTC m=+3161.693362254" Oct 30 19:38:48.210164 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:38:48.210120 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:38:56.088313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:38:56.088282 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:39:04.051994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:04.051960 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94"] Oct 30 19:39:04.097364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:04.097328 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eed18ff-chs94"] Oct 30 19:39:05.668926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:05.668890 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beaa48ce-e3f8-4a99-a2f0-7a25301cf681" path="/var/lib/kubelet/pods/beaa48ce-e3f8-4a99-a2f0-7a25301cf681/volumes" Oct 30 19:39:06.489690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:06.489655 4789 scope.go:122] "RemoveContainer" containerID="e6ab908acbb9c8074508dceaaf0081b8fa4e96ad01d5169f8a6aaaeea389e402" Oct 30 19:39:06.499495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:06.499473 4789 scope.go:122] "RemoveContainer" containerID="5ed15723f9c4ab8d52660ca362a7a12e14c2c3f890458f9cb51544580ced922a" Oct 30 19:39:08.118918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.118880 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:39:08.124577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.124554 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.200706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.200665 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-d6f58f\"" Oct 30 19:39:08.207088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.207055 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:39:08.231734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.228038 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-d6f58f-dockercfg-b475j\"" Oct 30 19:39:08.293167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.291532 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbg5\" (UniqueName: \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.293167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.291672 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.293167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.292698 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:39:08.297728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.297696 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.347511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.347476 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:39:08.392688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.392600 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpbh\" (UniqueName: \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.392847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.392710 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbg5\" (UniqueName: \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.392847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.392759 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.392847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.392834 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.393591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.393572 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.450980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.450950 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbg5\" (UniqueName: \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\") pod \"payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.493511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.493469 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpbh\" (UniqueName: \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.493707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.493553 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.494251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.494233 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.551845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.551814 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpbh\" (UniqueName: \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\") pod \"payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.612547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.612513 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:08.739950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.739920 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:08.993135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:08.993093 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:39:09.115428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.115382 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:39:09.116355 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:39:09.116332 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed8748b_7c88_4222_8225_51a52ab4c593.slice/crio-97cd84d7c520f2d16d671a3e7b1a91f9283cdddf3bc3599e00b43e378995996a WatchSource:0}: Error finding container 97cd84d7c520f2d16d671a3e7b1a91f9283cdddf3bc3599e00b43e378995996a: Status 404 returned error can't find the container with id 97cd84d7c520f2d16d671a3e7b1a91f9283cdddf3bc3599e00b43e378995996a Oct 30 19:39:09.306611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.306509 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" event={"ID":"9ed8748b-7c88-4222-8225-51a52ab4c593","Type":"ContainerStarted","Data":"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6"} Oct 30 19:39:09.306611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.306578 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" event={"ID":"9ed8748b-7c88-4222-8225-51a52ab4c593","Type":"ContainerStarted","Data":"97cd84d7c520f2d16d671a3e7b1a91f9283cdddf3bc3599e00b43e378995996a"} Oct 30 19:39:09.307108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.306655 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:09.308338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.308313 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" event={"ID":"898fd678-caa5-4550-8086-7ae7b848e63c","Type":"ContainerStarted","Data":"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189"} Oct 30 19:39:09.308480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.308358 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" event={"ID":"898fd678-caa5-4550-8086-7ae7b848e63c","Type":"ContainerStarted","Data":"eaa5ca79ba52ff78c0f6e3dbd71987b533428df7eedcea21e657a600c4c2e0b6"} Oct 30 19:39:09.308480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.308421 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:09.425134 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.425078 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" podStartSLOduration=1.42506428 podStartE2EDuration="1.42506428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:09.422061579 +0000 UTC m=+3185.923972065" watchObservedRunningTime="2026-10-30 19:39:09.42506428 +0000 UTC m=+3185.926974765" Oct 30 19:39:09.533456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:09.533401 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" podStartSLOduration=1.533386573 podStartE2EDuration="1.533386573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:09.531967941 +0000 UTC m=+3186.033878427" watchObservedRunningTime="2026-10-30 19:39:09.533386573 +0000 UTC m=+3186.035297081" Oct 30 19:39:15.051173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:15.051093 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm"] Oct 30 19:39:15.096851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:15.096821 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d286ebee-4whnm"] Oct 30 19:39:15.668907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:15.668875 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc14a5a6-2507-4eeb-9459-47af1d7abc0c" path="/var/lib/kubelet/pods/bc14a5a6-2507-4eeb-9459-47af1d7abc0c/volumes" Oct 30 19:39:18.197083 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:39:18.197047 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:39:20.322123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:20.322092 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:39:20.322566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:20.322161 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:39:29.471367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:29.471329 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:39:29.471866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:29.471411 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:39:30.411677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.411641 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:39:30.415081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.415058 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.471779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.471750 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:39:30.489259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.489193 4789 generic.go:356] "Generic (PLEG): container finished" podID="2c57b60a-be79-46a7-999b-639942a1c038" containerID="b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44" exitCode=0 Oct 30 19:39:30.489380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.489238 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerDied","Data":"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44"} Oct 30 19:39:30.489707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.489690 4789 scope.go:122] "RemoveContainer" containerID="b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44" Oct 30 19:39:30.491384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.491357 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:30.491501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.491435 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:30.491501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.491471 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:30.512489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.512470 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c01685\"" Oct 30 19:39:30.512637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.512546 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c01685-dockercfg-ktnlq\"" Oct 30 19:39:30.565422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.565174 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:39:30.580609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.579017 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.603175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.586587 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.603175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.586708 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gtnw\" (UniqueName: \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.670446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.670266 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:39:30.688122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.688079 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.688314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.688150 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wcn\" (UniqueName: \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.688314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.688262 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.688991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.688965 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.689130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.689107 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gtnw\" (UniqueName: \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.764764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.764727 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gtnw\" (UniqueName: \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\") pod \"payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.790201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.790162 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.790375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.790326 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wcn\" (UniqueName: \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.790920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.790892 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.794821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.794796 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:30.879532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.879509 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wcn\" (UniqueName: \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\") pod \"payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:30.903920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:30.903885 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:31.176461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.176435 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:39:31.178393 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:39:31.178367 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf299e81e_59da_447e_8d2d_b6260c50831e.slice/crio-c68932d404150c64d91125cd08d02c039a669c1cb8114fbddd1618c089fb4be3 WatchSource:0}: Error finding container c68932d404150c64d91125cd08d02c039a669c1cb8114fbddd1618c089fb4be3: Status 404 returned error can't find the container with id c68932d404150c64d91125cd08d02c039a669c1cb8114fbddd1618c089fb4be3 Oct 30 19:39:31.290397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.290362 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:39:31.320406 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:39:31.320366 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01327fa7_0b30_4de6_b800_3efb3b4d9d45.slice/crio-fc8ce93ea3e0a7bc4e9d24d8bcde702f13c6bc64ac8b196d304bd64c0d728f48 WatchSource:0}: Error finding container fc8ce93ea3e0a7bc4e9d24d8bcde702f13c6bc64ac8b196d304bd64c0d728f48: Status 404 returned error can't find the container with id fc8ce93ea3e0a7bc4e9d24d8bcde702f13c6bc64ac8b196d304bd64c0d728f48 Oct 30 19:39:31.520985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.520950 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerStarted","Data":"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f"} Oct 30 19:39:31.523353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.523324 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" event={"ID":"f299e81e-59da-447e-8d2d-b6260c50831e","Type":"ContainerStarted","Data":"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707"} Oct 30 19:39:31.523521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.523376 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" event={"ID":"f299e81e-59da-447e-8d2d-b6260c50831e","Type":"ContainerStarted","Data":"c68932d404150c64d91125cd08d02c039a669c1cb8114fbddd1618c089fb4be3"} Oct 30 19:39:31.523521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.523416 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:31.525140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.525115 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" event={"ID":"01327fa7-0b30-4de6-b800-3efb3b4d9d45","Type":"ContainerStarted","Data":"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474"} Oct 30 19:39:31.525293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.525152 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" event={"ID":"01327fa7-0b30-4de6-b800-3efb3b4d9d45","Type":"ContainerStarted","Data":"fc8ce93ea3e0a7bc4e9d24d8bcde702f13c6bc64ac8b196d304bd64c0d728f48"} Oct 30 19:39:31.525293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.525173 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:31.625624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.625591 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:39:31.625844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.625766 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:31.625844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.625804 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:31.751830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.751774 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" podStartSLOduration=1.751760978 podStartE2EDuration="1.751760978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:31.74739861 +0000 UTC m=+3208.249309097" watchObservedRunningTime="2026-10-30 19:39:31.751760978 +0000 UTC m=+3208.253671463" Oct 30 19:39:31.871153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:31.871044 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" podStartSLOduration=1.871023778 podStartE2EDuration="1.871023778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:31.868306867 +0000 UTC m=+3208.370217385" watchObservedRunningTime="2026-10-30 19:39:31.871023778 +0000 UTC m=+3208.372934265" Oct 30 19:39:32.626701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:32.626666 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:32.627192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:32.626720 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:33.627014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:33.626983 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:33.627425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:33.627036 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:34.626378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:34.626341 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:34.626575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:34.626402 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:35.626794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:35.626753 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:35.627256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:35.626861 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:36.626872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:36.626825 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 30 19:39:36.627303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:36.626896 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 30 19:39:37.630285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:37.630243 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:39:37.630780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:37.630636 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:39:37.631330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:37.631310 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:39:42.055948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:42.055913 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h"] Oct 30 19:39:42.102327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:42.102297 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-795aed21-9sw5h"] Oct 30 19:39:42.536544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:42.536517 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:39:42.536735 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:42.536586 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:39:43.669724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:43.669691 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccae1a1c-ec7b-4196-8793-5e47268af4c6" path="/var/lib/kubelet/pods/ccae1a1c-ec7b-4196-8793-5e47268af4c6/volumes" Oct 30 19:39:45.342724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.342691 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:39:45.348117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.348098 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.399006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.398968 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swztv\" (UniqueName: \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.399324 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.399301 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.415192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.415161 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:39:45.443308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.443277 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-59594f-dockercfg-fplzr\"" Oct 30 19:39:45.443608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.443589 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-59594f\"" Oct 30 19:39:45.494843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.494798 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:39:45.500855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.500808 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swztv\" (UniqueName: \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.501050 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.500912 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.501984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.501956 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.507472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.507447 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.551138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.551107 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:39:45.560129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.560101 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swztv\" (UniqueName: \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\") pod \"payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.602276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.602182 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.602447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.602427 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm2k9\" (UniqueName: \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.661660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.661632 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:45.704053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.703999 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.704255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.704193 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm2k9\" (UniqueName: \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.704921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.704893 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.762520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.762469 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm2k9\" (UniqueName: \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\") pod \"payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:45.824106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:45.824076 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:46.044352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.044322 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:39:46.045290 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:39:46.045264 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c5cc5f_a63f_4ce6_88d5_ee3236cc546f.slice/crio-38d86bd22c94440c299608d1f11848d0ef9d40c9350b36682fe9e1af29c618d5 WatchSource:0}: Error finding container 38d86bd22c94440c299608d1f11848d0ef9d40c9350b36682fe9e1af29c618d5: Status 404 returned error can't find the container with id 38d86bd22c94440c299608d1f11848d0ef9d40c9350b36682fe9e1af29c618d5 Oct 30 19:39:46.201045 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:39:46.201010 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80c1d37_3d0a_457a_93cc_ff28894122b4.slice/crio-0f362a91b2b4126113dcd5880a3ff56560aeb2d6be1de4e0a9d3a0505878274e WatchSource:0}: Error finding container 0f362a91b2b4126113dcd5880a3ff56560aeb2d6be1de4e0a9d3a0505878274e: Status 404 returned error can't find the container with id 0f362a91b2b4126113dcd5880a3ff56560aeb2d6be1de4e0a9d3a0505878274e Oct 30 19:39:46.202087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.202050 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:39:46.658921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.658816 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" event={"ID":"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f","Type":"ContainerStarted","Data":"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061"} Oct 30 19:39:46.658921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.658916 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" event={"ID":"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f","Type":"ContainerStarted","Data":"38d86bd22c94440c299608d1f11848d0ef9d40c9350b36682fe9e1af29c618d5"} Oct 30 19:39:46.659447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.658947 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:46.660680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.660653 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" event={"ID":"c80c1d37-3d0a-457a-93cc-ff28894122b4","Type":"ContainerStarted","Data":"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048"} Oct 30 19:39:46.660840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.660695 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" event={"ID":"c80c1d37-3d0a-457a-93cc-ff28894122b4","Type":"ContainerStarted","Data":"0f362a91b2b4126113dcd5880a3ff56560aeb2d6be1de4e0a9d3a0505878274e"} Oct 30 19:39:46.660840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.660773 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:46.778636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.778585 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" podStartSLOduration=1.778571423 podStartE2EDuration="1.778571423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:46.775542881 +0000 UTC m=+3223.277453366" watchObservedRunningTime="2026-10-30 19:39:46.778571423 +0000 UTC m=+3223.280481909" Oct 30 19:39:46.892868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:46.892807 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" podStartSLOduration=1.892787792 podStartE2EDuration="1.892787792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:46.885371844 +0000 UTC m=+3223.387282330" watchObservedRunningTime="2026-10-30 19:39:46.892787792 +0000 UTC m=+3223.394698279" Oct 30 19:39:48.197345 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:39:48.197296 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:39:57.670868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:57.670841 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:39:57.671269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:57.670900 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:39:59.005345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.005308 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": read tcp 10.132.0.2:33108->10.132.0.233:15021: read: connection reset by peer" start-of-body= Oct 30 19:39:59.005799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.005383 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": read tcp 10.132.0.2:33108->10.132.0.233:15021: read: connection reset by peer" Oct 30 19:39:59.782130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.782096 4789 generic.go:356] "Generic (PLEG): container finished" podID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerID="9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907" exitCode=0 Oct 30 19:39:59.782353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.782172 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerDied","Data":"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907"} Oct 30 19:39:59.782590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.782572 4789 scope.go:122] "RemoveContainer" containerID="9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907" Oct 30 19:39:59.784065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.784038 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:59.784163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.784101 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:39:59.784163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:39:59.784135 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:40:00.820348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:00.810872 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerStarted","Data":"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f"} Oct 30 19:40:01.261699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:01.261660 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:40:01.262010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:01.261983 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:40:01.262115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:01.262041 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:40:02.261888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:02.261853 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:40:02.262327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:02.261910 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:40:03.261280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:03.261249 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:40:03.261461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:03.261303 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:40:04.261527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:04.261496 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:40:04.261909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:04.261552 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:40:05.261612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:05.261570 4789 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:40:05.262018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:05.261653 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:40:06.266052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.266019 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:40:06.266568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.266491 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:40:06.267093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.267074 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:40:06.531349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.531256 4789 scope.go:122] "RemoveContainer" containerID="09c880094cc7484dc8784d3dde8429c0ab05d5e3f243cfe575d46e9d7538fc42" Oct 30 19:40:06.544141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.544111 4789 scope.go:122] "RemoveContainer" containerID="592d30070d420f59dfac5366e9451b88d974005bf1a3f87ac9a8e1572de3022c" Oct 30 19:40:06.556462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.556437 4789 scope.go:122] "RemoveContainer" containerID="21e87c16e443e9029c3781487348eb1a3a814bfa50a3d2a67ad2404a29bd2e12" Oct 30 19:40:06.574239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.574183 4789 scope.go:122] "RemoveContainer" containerID="96e92f04f3ab8d93717ec10f465490ee2f0c076e254f131fd279ecff245d146b" Oct 30 19:40:06.587659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:06.587627 4789 scope.go:122] "RemoveContainer" containerID="e8253a1e2a85bffb778a5aaa3db1d1c26e7b8ee7474482f08fad106a1369df5b" Oct 30 19:40:14.350440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.350410 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:40:14.356951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.356931 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.382980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.382951 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.383141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.383120 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9cvk\" (UniqueName: \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.406233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.406183 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a2f378\"" Oct 30 19:40:14.410803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.410780 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:40:14.470714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.470567 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a2f378-dockercfg-4w4fl\"" Oct 30 19:40:14.486687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.486646 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.486897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.486833 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9cvk\" (UniqueName: \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.487914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.487839 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.527361 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.527322 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:40:14.548352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.537415 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.548352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.548197 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9cvk\" (UniqueName: \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\") pod \"payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.566260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.565350 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:40:14.587857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.587819 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5bd\" (UniqueName: \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.587993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.587975 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.670023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.669954 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:14.689273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.689246 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5bd\" (UniqueName: \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.689408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.689323 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.690376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.690302 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.751093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.751068 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5bd\" (UniqueName: \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\") pod \"payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:14.879364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:14.879332 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:15.046765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.046738 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:40:15.047866 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:40:15.047836 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77819d6d_5ced_4dac_b0c6_2fc83d4fd642.slice/crio-33e3711231ca06ddafc23757d01cb8cd2b32c40a9a2201081ef4d932c751d4f2 WatchSource:0}: Error finding container 33e3711231ca06ddafc23757d01cb8cd2b32c40a9a2201081ef4d932c751d4f2: Status 404 returned error can't find the container with id 33e3711231ca06ddafc23757d01cb8cd2b32c40a9a2201081ef4d932c751d4f2 Oct 30 19:40:15.053540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.053521 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb"] Oct 30 19:40:15.056789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.056767 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c3b3f9-6lgjb"] Oct 30 19:40:15.059740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.059714 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5"] Oct 30 19:40:15.062795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.062773 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71d9477a-68rz5"] Oct 30 19:40:15.260545 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:40:15.260512 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2c5ae7_7ea6_4527_943a_34bc7d34efea.slice/crio-66ef110aa9d305159146dddeb0e8e4b9b7764f80a974735abc5efb9507cb98a1 WatchSource:0}: Error finding container 66ef110aa9d305159146dddeb0e8e4b9b7764f80a974735abc5efb9507cb98a1: Status 404 returned error can't find the container with id 66ef110aa9d305159146dddeb0e8e4b9b7764f80a974735abc5efb9507cb98a1 Oct 30 19:40:15.261463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.261425 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:40:15.669539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.669461 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45682fdf-06d1-476e-8e5d-750cf4a32b4e" path="/var/lib/kubelet/pods/45682fdf-06d1-476e-8e5d-750cf4a32b4e/volumes" Oct 30 19:40:15.670037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.670021 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c58ce6b-2074-478d-8400-970134c9c081" path="/var/lib/kubelet/pods/8c58ce6b-2074-478d-8400-970134c9c081/volumes" Oct 30 19:40:15.942912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.942875 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" event={"ID":"ee2c5ae7-7ea6-4527-943a-34bc7d34efea","Type":"ContainerStarted","Data":"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023"} Oct 30 19:40:15.943094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.942933 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" event={"ID":"ee2c5ae7-7ea6-4527-943a-34bc7d34efea","Type":"ContainerStarted","Data":"66ef110aa9d305159146dddeb0e8e4b9b7764f80a974735abc5efb9507cb98a1"} Oct 30 19:40:15.943094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.942965 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:15.944604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.944581 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" event={"ID":"77819d6d-5ced-4dac-b0c6-2fc83d4fd642","Type":"ContainerStarted","Data":"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69"} Oct 30 19:40:15.944731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.944617 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" event={"ID":"77819d6d-5ced-4dac-b0c6-2fc83d4fd642","Type":"ContainerStarted","Data":"33e3711231ca06ddafc23757d01cb8cd2b32c40a9a2201081ef4d932c751d4f2"} Oct 30 19:40:15.944731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:15.944672 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:16.056082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:16.056030 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" podStartSLOduration=2.05601554 podStartE2EDuration="2.05601554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:16.053360311 +0000 UTC m=+3252.555270797" watchObservedRunningTime="2026-10-30 19:40:16.05601554 +0000 UTC m=+3252.557926025" Oct 30 19:40:16.165427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:16.165370 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" podStartSLOduration=2.165353572 podStartE2EDuration="2.165353572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:16.163612062 +0000 UTC m=+3252.665522548" watchObservedRunningTime="2026-10-30 19:40:16.165353572 +0000 UTC m=+3252.667264053" Oct 30 19:40:18.197404 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:40:18.197370 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:40:26.053445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:26.053413 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg"] Oct 30 19:40:26.099448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:26.099418 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d0c0db-n9wpg"] Oct 30 19:40:26.955131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:26.955102 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:40:26.955309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:26.955164 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:40:27.669000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:27.668962 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bd2536-fd1c-4345-aa1a-865505c86cd6" path="/var/lib/kubelet/pods/46bd2536-fd1c-4345-aa1a-865505c86cd6/volumes" Oct 30 19:40:29.057730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:29.057693 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn"] Oct 30 19:40:29.101670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:29.101640 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d9b39e-dj6dn"] Oct 30 19:40:29.671754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:29.671663 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6327c370-f397-4af2-baab-7f33b76407f5" path="/var/lib/kubelet/pods/6327c370-f397-4af2-baab-7f33b76407f5/volumes" Oct 30 19:40:33.052405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:33.052376 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc"] Oct 30 19:40:33.098090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:33.098059 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4756d9bd-hq6bc"] Oct 30 19:40:33.670203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:33.670169 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37afaa6-a08f-46ca-9081-4b4826696776" path="/var/lib/kubelet/pods/e37afaa6-a08f-46ca-9081-4b4826696776/volumes" Oct 30 19:40:43.060407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:43.060152 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98"] Oct 30 19:40:43.107616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:43.107581 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3444fd80-ckn98"] Oct 30 19:40:43.668710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:43.668678 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39fa37d-b715-47ea-9bef-5df2251a01ab" path="/var/lib/kubelet/pods/b39fa37d-b715-47ea-9bef-5df2251a01ab/volumes" Oct 30 19:40:48.199042 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:40:48.199009 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:40:51.309572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.309536 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:40:51.316332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.316308 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.367711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.367674 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:40:51.404602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.404570 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.404781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.404607 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.404865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.404840 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.404947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.404933 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.405020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.405001 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.405077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.405062 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78986\" (UniqueName: \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.408889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.408869 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-db06cf\"/\"auth-test-model-db06cf-kserve-self-signed-certs\"" Oct 30 19:40:51.409006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.408926 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-db06cf\"/\"default-dockercfg-tcvwn\"" Oct 30 19:40:51.411272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.411250 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-db06cf\"/\"kube-root-ca.crt\"" Oct 30 19:40:51.411345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.411294 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-db06cf\"/\"openshift-service-ca.crt\"" Oct 30 19:40:51.506821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.506781 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.506836 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.506989 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.507029 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.507118 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.507164 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78986\" (UniqueName: \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.507864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.507819 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.510576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.510549 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.511405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.511377 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.511524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.511242 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.512776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.512752 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.592247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.592146 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78986\" (UniqueName: \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\") pod \"auth-test-model-db06cf-kserve-6cc4f4844f-gcthm\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:51.630521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:51.630495 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:52.014738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:52.014710 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:40:52.017148 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:40:52.017119 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3db2e92_703f_43ff_be8b_a32ef4310498.slice/crio-895463c20c6fceee2fc53fd5ec0bfc75a94ab4fdcccfb1f42e728c671c0f7516 WatchSource:0}: Error finding container 895463c20c6fceee2fc53fd5ec0bfc75a94ab4fdcccfb1f42e728c671c0f7516: Status 404 returned error can't find the container with id 895463c20c6fceee2fc53fd5ec0bfc75a94ab4fdcccfb1f42e728c671c0f7516 Oct 30 19:40:52.239220 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:52.239173 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" event={"ID":"f3db2e92-703f-43ff-be8b-a32ef4310498","Type":"ContainerStarted","Data":"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95"} Oct 30 19:40:52.239387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:52.239232 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" event={"ID":"f3db2e92-703f-43ff-be8b-a32ef4310498","Type":"ContainerStarted","Data":"895463c20c6fceee2fc53fd5ec0bfc75a94ab4fdcccfb1f42e728c671c0f7516"} Oct 30 19:40:52.353882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:52.353795 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" podStartSLOduration=1.353781576 podStartE2EDuration="1.353781576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:52.349949845 +0000 UTC m=+3288.851860333" watchObservedRunningTime="2026-10-30 19:40:52.353781576 +0000 UTC m=+3288.855692062" Oct 30 19:40:52.631243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:52.631151 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:40:59.052514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:59.052475 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp"] Oct 30 19:40:59.098072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:59.098036 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9f5cf04-jqhjp"] Oct 30 19:40:59.669015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:40:59.668976 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6bfafb-f405-4508-8bb3-87b850a8b786" path="/var/lib/kubelet/pods/dd6bfafb-f405-4508-8bb3-87b850a8b786/volumes" Oct 30 19:41:01.631190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:01.631155 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:41:01.643782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:01.643758 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:41:02.059667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:02.059629 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr"] Oct 30 19:41:02.062944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:02.062901 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c7c1a48-l8mjr"] Oct 30 19:41:02.323484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:02.323401 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:41:03.670891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:03.670854 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4646599-4d53-4750-83bf-e042ed4e7a88" path="/var/lib/kubelet/pods/a4646599-4d53-4750-83bf-e042ed4e7a88/volumes" Oct 30 19:41:04.483601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:04.483573 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:41:04.490482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:04.490455 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:41:06.057934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.057900 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5"] Oct 30 19:41:06.060993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.060965 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1c7991c-hskb5"] Oct 30 19:41:06.664347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.664320 4789 scope.go:122] "RemoveContainer" containerID="56c883f76d79a4aa7e9ad254eb8a56203a035db3522e8095257850dcd3628ba6" Oct 30 19:41:06.674283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.674263 4789 scope.go:122] "RemoveContainer" containerID="76376c897275a3d3c49962b4c1f1b1b0c5df85823ed8e3ec36c06986df307cfc" Oct 30 19:41:06.683440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.683418 4789 scope.go:122] "RemoveContainer" containerID="e84e72f705d271c856cb68251d559b427c3f3ce5e79f54d1e5a7e934e828afae" Oct 30 19:41:06.693355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.693332 4789 scope.go:122] "RemoveContainer" containerID="a1670c13ea2b69e7ea14f2e5cdaa31a317cba5620bf7e47b41e7122339ef8c26" Oct 30 19:41:06.702738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:06.702716 4789 scope.go:122] "RemoveContainer" containerID="3f4efcf56bf935413c6d04652ca3eb2436db3d189061f0f57b2e45f8337750c8" Oct 30 19:41:07.670095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:07.670051 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ffccdc-1087-478e-80fd-7077f201326d" path="/var/lib/kubelet/pods/51ffccdc-1087-478e-80fd-7077f201326d/volumes" Oct 30 19:41:11.049599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:11.049567 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh"] Oct 30 19:41:11.095048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:11.095016 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1517eb93-rnflh"] Oct 30 19:41:11.669067 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:11.669031 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20770d7-a16a-48f9-8416-64ec1af203a1" path="/var/lib/kubelet/pods/a20770d7-a16a-48f9-8416-64ec1af203a1/volumes" Oct 30 19:41:18.196974 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:41:18.196929 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:41:21.052577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:21.052543 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w"] Oct 30 19:41:21.098526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:21.098497 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b41acf4-j296w"] Oct 30 19:41:21.679670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:21.679623 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1" path="/var/lib/kubelet/pods/ca2ddc4c-1ea3-4b91-bc81-0db58328b4f1/volumes" Oct 30 19:41:45.055394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.055357 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx"] Oct 30 19:41:45.102372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.102328 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799"] Oct 30 19:41:45.105665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.105636 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a26a37-qsxxx"] Oct 30 19:41:45.108325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.108302 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b4284b7-t8799"] Oct 30 19:41:45.668531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.668501 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b99ced-4252-4e9f-b941-73635f87cd13" path="/var/lib/kubelet/pods/28b99ced-4252-4e9f-b941-73635f87cd13/volumes" Oct 30 19:41:45.668976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:45.668960 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b189f7-8f15-4b3d-9fde-0d88a566ade4" path="/var/lib/kubelet/pods/30b189f7-8f15-4b3d-9fde-0d88a566ade4/volumes" Oct 30 19:41:48.197772 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:41:48.197729 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:41:53.017121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.017088 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:41:53.022483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.022466 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.092890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.092855 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2186d8\"/\"openshift-service-ca.crt\"" Oct 30 19:41:53.120138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.120099 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2186d8\"/\"auth-test-model-2186d8-kserve-self-signed-certs\"" Oct 30 19:41:53.120335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.120168 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2186d8\"/\"kube-root-ca.crt\"" Oct 30 19:41:53.120335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.120168 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2186d8\"/\"default-dockercfg-xd8h2\"" Oct 30 19:41:53.123644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.123620 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:41:53.128732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.128711 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.128849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.128760 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.128937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.128917 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dbr\" (UniqueName: \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.129001 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.128974 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.129077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.129062 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.129140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.129124 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230182 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230335 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dbr\" (UniqueName: \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230384 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230444 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230535 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230894 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.230929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230921 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.231148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.230977 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.232816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.232788 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.233009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.232988 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.291142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.291070 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dbr\" (UniqueName: \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\") pod \"auth-test-model-2186d8-kserve-64f6d95c45-qqjmj\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.336709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.336680 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:53.718075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:53.718047 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:41:53.720227 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:41:53.720183 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4e4e6e_07cd_4a3f_a2ed_f4fe62d02fb8.slice/crio-ed545f643936a8ae06d78ffb09846683de8ecb3ea79dc2b61d5fb2f11d33f151 WatchSource:0}: Error finding container ed545f643936a8ae06d78ffb09846683de8ecb3ea79dc2b61d5fb2f11d33f151: Status 404 returned error can't find the container with id ed545f643936a8ae06d78ffb09846683de8ecb3ea79dc2b61d5fb2f11d33f151 Oct 30 19:41:54.720076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:54.720041 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" event={"ID":"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8","Type":"ContainerStarted","Data":"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e"} Oct 30 19:41:54.720505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:54.720084 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" event={"ID":"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8","Type":"ContainerStarted","Data":"ed545f643936a8ae06d78ffb09846683de8ecb3ea79dc2b61d5fb2f11d33f151"} Oct 30 19:41:54.830849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:54.830790 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" podStartSLOduration=2.830773076 podStartE2EDuration="2.830773076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:54.830416158 +0000 UTC m=+3351.332326644" watchObservedRunningTime="2026-10-30 19:41:54.830773076 +0000 UTC m=+3351.332683563" Oct 30 19:41:55.089471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:55.089388 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5"] Oct 30 19:41:55.099761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:55.099730 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd507597-mzkn5"] Oct 30 19:41:55.336883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:55.336855 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:41:55.671702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:41:55.671670 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee77beb-32a5-40eb-88ff-1dec4f59147e" path="/var/lib/kubelet/pods/aee77beb-32a5-40eb-88ff-1dec4f59147e/volumes" Oct 30 19:42:03.337094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:03.337054 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:42:03.349463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:03.349433 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:42:03.802235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:03.802183 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:42:05.050802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:05.050766 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v"] Oct 30 19:42:05.095974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:05.095943 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f264a3-4bb8v"] Oct 30 19:42:05.668998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:05.668963 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8d456b-8657-4576-97de-a57bbc104293" path="/var/lib/kubelet/pods/5a8d456b-8657-4576-97de-a57bbc104293/volumes" Oct 30 19:42:06.757673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.757639 4789 scope.go:122] "RemoveContainer" containerID="b5acf552a088a59130ee9e53f675e322074ef62bbe5f850cc068d95ce99b32fd" Oct 30 19:42:06.768199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.768173 4789 scope.go:122] "RemoveContainer" containerID="aba5b21b984fc0734c11d53bfe7a76442b5bd5669f5346e24ca8eba65e694b82" Oct 30 19:42:06.777958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.777940 4789 scope.go:122] "RemoveContainer" containerID="fa074e57d93759ea19cf82416c77b30c51da6aaf2e38130ea279198fca350d6b" Oct 30 19:42:06.786765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.786745 4789 scope.go:122] "RemoveContainer" containerID="afa1f11ee76fad693bb0d0b94bfc998bb31ec59aa0e72a3764a9e2236dfae07a" Oct 30 19:42:06.796272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.796251 4789 scope.go:122] "RemoveContainer" containerID="37ff196a9e3c255c4db500b6011c5927d65b0bc7a5f1dd9df22505ee04f164d6" Oct 30 19:42:06.807326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.807309 4789 scope.go:122] "RemoveContainer" containerID="26b5797685b4b6c02451c1e59b59e1f465255c821c5b25b352bfca9fa96c1a1d" Oct 30 19:42:06.819104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.819076 4789 scope.go:122] "RemoveContainer" containerID="c7f4607dd52f59810a97f9f61432b716684d6148626cc070d1a160d5d7cdbee8" Oct 30 19:42:06.828380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.828361 4789 scope.go:122] "RemoveContainer" containerID="a2f46efb57de2a5c1d283cc4ecb25f7f20345bb78de0a2c932fd19058043136d" Oct 30 19:42:06.836732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:06.836716 4789 scope.go:122] "RemoveContainer" containerID="3445accc787857a3610331831000542b5c447c0c7b74bffa06ac282aff78f81d" Oct 30 19:42:09.233527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:09.233498 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:42:09.233982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:09.233730 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" containerID="cri-o://7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6" gracePeriod=30 Oct 30 19:42:11.050371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:11.050323 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94"] Oct 30 19:42:11.095787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:11.095756 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341fa872-k2m94"] Oct 30 19:42:11.670377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:11.670335 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c662fb6e-1fa0-4c7c-9024-a7d072b870c9" path="/var/lib/kubelet/pods/c662fb6e-1fa0-4c7c-9024-a7d072b870c9/volumes" Oct 30 19:42:14.516252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.516151 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:42:14.594561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594526 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.594770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594577 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.594770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594625 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.594770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594727 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.594964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594769 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.594964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594813 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:14.594964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594910 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.595137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.594972 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.595137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595005 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.595137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595065 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\") pod \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\" (UID: \"1949abeb-e731-4bb4-8351-a9e2e80ac9fc\") " Oct 30 19:42:14.595396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595065 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:14.595396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595072 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:14.595396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595275 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:14.595655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595533 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:14.595655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595642 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.595777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595665 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.595777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595679 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.595777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595695 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.595777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.595708 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.597686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.597661 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:14.597814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.597745 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "kube-api-access-5cx2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:14.600563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.600523 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:14.600759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.600738 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy" pod "1949abeb-e731-4bb4-8351-a9e2e80ac9fc" (UID: "1949abeb-e731-4bb4-8351-a9e2e80ac9fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:14.697437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.697399 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.697437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.697437 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.697746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.697451 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.697746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.697465 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cx2r\" (UniqueName: \"kubernetes.io/projected/1949abeb-e731-4bb4-8351-a9e2e80ac9fc-kube-api-access-5cx2r\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:14.897169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.897075 4789 generic.go:356] "Generic (PLEG): container finished" podID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerID="7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6" exitCode=0 Oct 30 19:42:14.897381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.897285 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" event={"ID":"1949abeb-e731-4bb4-8351-a9e2e80ac9fc","Type":"ContainerDied","Data":"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6"} Oct 30 19:42:14.897381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.897340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" event={"ID":"1949abeb-e731-4bb4-8351-a9e2e80ac9fc","Type":"ContainerDied","Data":"ff8956dc81350d6ab66ae54d10da5a2745b65c67120880a5d98fe99b44cefff8"} Oct 30 19:42:14.897381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.897368 4789 scope.go:122] "RemoveContainer" containerID="7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6" Oct 30 19:42:14.897566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.897527 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp" Oct 30 19:42:14.913865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.913810 4789 scope.go:122] "RemoveContainer" containerID="7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6" Oct 30 19:42:14.914260 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:42:14.914199 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6\": container with ID starting with 7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6 not found: ID does not exist" containerID="7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6" Oct 30 19:42:14.914398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:14.914269 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6"} err="failed to get container status \"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6\": rpc error: code = NotFound desc = could not find container \"7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6\": container with ID starting with 7262afec40f7610009c71a30ba9fcc8587367bf59c766d1d11b038954367c4c6 not found: ID does not exist" Oct 30 19:42:15.059363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:15.059325 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:42:15.063152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:15.063113 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-hgnjp"] Oct 30 19:42:15.669132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:15.669096 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" path="/var/lib/kubelet/pods/1949abeb-e731-4bb4-8351-a9e2e80ac9fc/volumes" Oct 30 19:42:18.196910 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:42:18.196873 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:42:45.162604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:45.162570 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:42:45.163265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:45.162817 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" containerID="cri-o://98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea" gracePeriod=30 Oct 30 19:42:45.208719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:45.208689 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:42:45.208974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:45.208919 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" containerID="cri-o://9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd" gracePeriod=30 Oct 30 19:42:48.197723 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:42:48.197687 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:42:50.498473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.498445 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:42:50.516681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516650 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.516847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516699 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.516847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516737 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.516847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516780 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.516847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516814 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.517079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516874 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.517079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516939 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.517079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.516968 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.517079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.517073 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.517956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.517167 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.517956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.517188 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\") pod \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\" (UID: \"fa515561-b454-4d60-ab83-a38a8cf3c3b7\") " Oct 30 19:42:50.517956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.517679 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.518136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.518007 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.518136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.518030 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.518136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.518044 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.518348 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.518323 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.518576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.518552 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:50.519955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.519905 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.520526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.520493 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:50.520633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.520555 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:50.520859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.520839 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k" pod "fa515561-b454-4d60-ab83-a38a8cf3c3b7" (UID: "fa515561-b454-4d60-ab83-a38a8cf3c3b7"). InnerVolumeSpecName "kube-api-access-qrn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:50.550179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.550158 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:42:50.618895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.618872 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.618996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.618917 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619017 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619048 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619086 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619119 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619158 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619203 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619154 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619266 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\") pod \"bb63e789-972b-49a4-a71e-a96b78304dcb\" (UID: \"bb63e789-972b-49a4-a71e-a96b78304dcb\") " Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619272 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.619500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619425 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619518 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619733 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619755 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619771 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619786 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619801 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619815 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619829 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619842 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrn9k\" (UniqueName: \"kubernetes.io/projected/fa515561-b454-4d60-ab83-a38a8cf3c3b7-kube-api-access-qrn9k\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619857 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619870 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa515561-b454-4d60-ab83-a38a8cf3c3b7-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.619880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.619876 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:50.621535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.621515 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "kube-api-access-hkcg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:50.621712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.621692 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:50.621790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.621720 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:42:50.622656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.622640 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo" pod "bb63e789-972b-49a4-a71e-a96b78304dcb" (UID: "bb63e789-972b-49a4-a71e-a96b78304dcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:42:50.721477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.721394 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcg4\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-kube-api-access-hkcg4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.721477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.721423 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.721477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.721437 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.721477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.721450 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb63e789-972b-49a4-a71e-a96b78304dcb-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:50.721477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:50.721462 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb63e789-972b-49a4-a71e-a96b78304dcb-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:51.158102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.158013 4789 generic.go:356] "Generic (PLEG): container finished" podID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerID="98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea" exitCode=0 Oct 30 19:42:51.158102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.158065 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" Oct 30 19:42:51.158358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.158096 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" event={"ID":"fa515561-b454-4d60-ab83-a38a8cf3c3b7","Type":"ContainerDied","Data":"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea"} Oct 30 19:42:51.158358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.158150 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq" event={"ID":"fa515561-b454-4d60-ab83-a38a8cf3c3b7","Type":"ContainerDied","Data":"bab1cf99dcf6252240a426127e75276ed0c5cdf8c22146e1f42f1061cc366bcc"} Oct 30 19:42:51.158358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.158173 4789 scope.go:122] "RemoveContainer" containerID="98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea" Oct 30 19:42:51.159616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.159588 4789 generic.go:356] "Generic (PLEG): container finished" podID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerID="9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd" exitCode=0 Oct 30 19:42:51.159721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.159616 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" event={"ID":"bb63e789-972b-49a4-a71e-a96b78304dcb","Type":"ContainerDied","Data":"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd"} Oct 30 19:42:51.159721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.159665 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" Oct 30 19:42:51.159835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.159666 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls" event={"ID":"bb63e789-972b-49a4-a71e-a96b78304dcb","Type":"ContainerDied","Data":"acd7b0b866be5b5423c2933aa9129a51123807e270091201d48a27eaf8165072"} Oct 30 19:42:51.169462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.169444 4789 scope.go:122] "RemoveContainer" containerID="98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea" Oct 30 19:42:51.172447 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:42:51.172418 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea\": container with ID starting with 98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea not found: ID does not exist" containerID="98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea" Oct 30 19:42:51.172535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.172456 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea"} err="failed to get container status \"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea\": rpc error: code = NotFound desc = could not find container \"98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea\": container with ID starting with 98ec6d48c4fe21509d7213b94d8ad52d48fb43ee5995b512d863a4263c575eea not found: ID does not exist" Oct 30 19:42:51.172535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.172480 4789 scope.go:122] "RemoveContainer" containerID="9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd" Oct 30 19:42:51.181733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.181717 4789 scope.go:122] "RemoveContainer" containerID="9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd" Oct 30 19:42:51.182013 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:42:51.181994 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd\": container with ID starting with 9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd not found: ID does not exist" containerID="9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd" Oct 30 19:42:51.182085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.182017 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd"} err="failed to get container status \"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd\": rpc error: code = NotFound desc = could not find container \"9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd\": container with ID starting with 9c01a8ae3ef9ad1263a6b0e617800083e5cdedaded0b377b3170e7c1e6fa09cd not found: ID does not exist" Oct 30 19:42:51.321052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.321021 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:42:51.365765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.365735 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-6lcsq"] Oct 30 19:42:51.469197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.469162 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:42:51.473451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.473424 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-22fb18-openshift-default-d9c87fb8f-8l2ls"] Oct 30 19:42:51.669339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.669310 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" path="/var/lib/kubelet/pods/bb63e789-972b-49a4-a71e-a96b78304dcb/volumes" Oct 30 19:42:51.669817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:51.669801 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" path="/var/lib/kubelet/pods/fa515561-b454-4d60-ab83-a38a8cf3c3b7/volumes" Oct 30 19:42:58.734420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.734379 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:42:58.735242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735221 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" Oct 30 19:42:58.735316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735244 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" Oct 30 19:42:58.735316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735281 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" Oct 30 19:42:58.735316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735289 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" Oct 30 19:42:58.735316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735305 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" Oct 30 19:42:58.735316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735312 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" Oct 30 19:42:58.735510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735478 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa515561-b454-4d60-ab83-a38a8cf3c3b7" containerName="istio-proxy" Oct 30 19:42:58.735510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735493 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb63e789-972b-49a4-a71e-a96b78304dcb" containerName="istio-proxy" Oct 30 19:42:58.735584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.735513 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1949abeb-e731-4bb4-8351-a9e2e80ac9fc" containerName="istio-proxy" Oct 30 19:42:58.740962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.740942 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.807673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807640 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.807838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807692 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5dg\" (UniqueName: \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.807838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807769 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.807838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807785 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.807838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807834 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.808097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.807867 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.836863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.836834 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2604c\"/\"auth-test-model-b2604c-kserve-self-signed-certs\"" Oct 30 19:42:58.836863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.836861 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2604c\"/\"default-dockercfg-cqm4h\"" Oct 30 19:42:58.837083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.836836 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2604c\"/\"openshift-service-ca.crt\"" Oct 30 19:42:58.837083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.836838 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b2604c\"/\"kube-root-ca.crt\"" Oct 30 19:42:58.841739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.841719 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909411 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909477 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909563 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909611 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5dg\" (UniqueName: \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909725 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.909751 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.910917 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.911242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.911157 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.912974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.912934 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.913203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.913063 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.913372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.913179 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:58.974203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:58.974167 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5dg\" (UniqueName: \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\") pod \"auth-test-model-b2604c-kserve-f5f8dbc7-tq548\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:59.053083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:59.052983 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:42:59.467107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:42:59.467078 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:42:59.469608 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:42:59.469584 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08c6825a_2fb4_45b1_b7ff_6e3c91dc50d6.slice/crio-d4d8f8ab3c0ade250e0e99db8b30452ab2d7d5c9b36ed5348406aac4eaae834b WatchSource:0}: Error finding container d4d8f8ab3c0ade250e0e99db8b30452ab2d7d5c9b36ed5348406aac4eaae834b: Status 404 returned error can't find the container with id d4d8f8ab3c0ade250e0e99db8b30452ab2d7d5c9b36ed5348406aac4eaae834b Oct 30 19:43:00.268428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:00.268388 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" event={"ID":"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6","Type":"ContainerStarted","Data":"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c"} Oct 30 19:43:00.268840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:00.268442 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" event={"ID":"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6","Type":"ContainerStarted","Data":"d4d8f8ab3c0ade250e0e99db8b30452ab2d7d5c9b36ed5348406aac4eaae834b"} Oct 30 19:43:00.381946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:00.381895 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" podStartSLOduration=2.381881461 podStartE2EDuration="2.381881461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:00.378890075 +0000 UTC m=+3416.880800562" watchObservedRunningTime="2026-10-30 19:43:00.381881461 +0000 UTC m=+3416.883791972" Oct 30 19:43:00.427949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:00.427919 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:43:00.428131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:00.428108 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" containerID="cri-o://03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178" gracePeriod=30 Oct 30 19:43:01.053553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:01.053520 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:43:05.704916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.704848 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:43:05.810182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810147 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810435 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810490 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810578 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810695 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810722 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.810770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810744 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810814 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810844 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.810981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.810938 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\") pod \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\" (UID: \"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf\") " Oct 30 19:43:05.811035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.811016 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.813465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.813242 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:05.813673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.813619 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.813974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.813947 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.820052 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "kube-api-access-nfxt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.820639 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.820718 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821007 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token" pod "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" (UID: "1c1ab8e8-001c-4080-a741-fc88b4ce6fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821186 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821314 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821330 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821344 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821360 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821373 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821387 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821401 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:05.832247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:05.821414 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxt4\" (UniqueName: \"kubernetes.io/projected/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf-kube-api-access-nfxt4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:06.314872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.314833 4789 generic.go:356] "Generic (PLEG): container finished" podID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerID="03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178" exitCode=0 Oct 30 19:43:06.315090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.314880 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" Oct 30 19:43:06.315090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.314929 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" event={"ID":"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf","Type":"ContainerDied","Data":"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178"} Oct 30 19:43:06.315090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.314996 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22" event={"ID":"1c1ab8e8-001c-4080-a741-fc88b4ce6fbf","Type":"ContainerDied","Data":"99d0b4ed1c097fe3ba662b6c2bfe6a5d98d233a18414c26ed88718b0cbade5f0"} Oct 30 19:43:06.315090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.315024 4789 scope.go:122] "RemoveContainer" containerID="03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178" Oct 30 19:43:06.329777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.329756 4789 scope.go:122] "RemoveContainer" containerID="03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178" Oct 30 19:43:06.330128 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:43:06.330094 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178\": container with ID starting with 03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178 not found: ID does not exist" containerID="03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178" Oct 30 19:43:06.330189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.330141 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178"} err="failed to get container status \"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178\": rpc error: code = NotFound desc = could not find container \"03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178\": container with ID starting with 03f04f5eade08e608b353c3451e420fc200e5c8785f7c7a4fd48236e88ebe178 not found: ID does not exist" Oct 30 19:43:06.474750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.474624 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:43:06.519236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.519185 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-5zm22"] Oct 30 19:43:06.903091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.903057 4789 scope.go:122] "RemoveContainer" containerID="4f4b6ae5c40d6d0b3e9e9846a183b6b0794eca9da2cb004565e690a796748c4a" Oct 30 19:43:06.913339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.913319 4789 scope.go:122] "RemoveContainer" containerID="e46b613ea16b3730ecbce602ec8f92f7d04e613950a173c4737682967fd81dcb" Oct 30 19:43:06.924444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.924425 4789 scope.go:122] "RemoveContainer" containerID="71df29559153f8e1a60851f02b7fa0d15e4b77080f51d8c02a5195cda79f71fb" Oct 30 19:43:06.935172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.935150 4789 scope.go:122] "RemoveContainer" containerID="bacd35b0d1becec43338aa5aadda4445b3c3a4587f1d6734083b0e85b8b1c402" Oct 30 19:43:06.945006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.944981 4789 scope.go:122] "RemoveContainer" containerID="2d424d4fe8ccbdd8ffe8eaf340270903a87a1414f8beef66c4b0310bffd60ac5" Oct 30 19:43:06.952987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.952964 4789 scope.go:122] "RemoveContainer" containerID="77ed668160beba18abbc239531da47764b134b43383d1453d6d0c18c7a8719f0" Oct 30 19:43:06.961287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.961267 4789 scope.go:122] "RemoveContainer" containerID="3405e419b15f6d774d98fe1c3a6768db6234e9346d42b8c8d372908a2ac7115a" Oct 30 19:43:06.969334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:06.969315 4789 scope.go:122] "RemoveContainer" containerID="77462bf5c1f74444506daa85809e2bc7485138b0ccaddf880a1f9df33a61e46b" Oct 30 19:43:07.668686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:07.668651 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" path="/var/lib/kubelet/pods/1c1ab8e8-001c-4080-a741-fc88b4ce6fbf/volumes" Oct 30 19:43:09.053946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:09.053915 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:43:09.066725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:09.066702 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:43:09.346586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:09.346508 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:43:18.198050 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:43:18.198010 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:43:48.196982 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:43:48.196900 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:43:51.575561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.575525 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:43:51.576241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.576205 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" Oct 30 19:43:51.576294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.576243 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" Oct 30 19:43:51.576391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.576381 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1ab8e8-001c-4080-a741-fc88b4ce6fbf" containerName="istio-proxy" Oct 30 19:43:51.579888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.579871 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636399 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636492 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfdv\" (UniqueName: \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636532 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636599 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636688 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.636799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.636719 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.650143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.650113 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:43:51.677723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.677699 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-af72cf\"/\"default-dockercfg-7p8wv\"" Oct 30 19:43:51.677867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.677737 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-af72cf\"/\"openshift-service-ca.crt\"" Oct 30 19:43:51.677867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.677778 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-af72cf\"/\"kube-root-ca.crt\"" Oct 30 19:43:51.677867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.677838 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-af72cf\"/\"praxis-route-af72cf-kserve-self-signed-certs\"" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738058 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738156 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfdv\" (UniqueName: \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738199 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738304 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738399 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.738435 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.739417 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.739623 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.740104 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.741747 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.742082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.741916 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.799097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.799063 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfdv\" (UniqueName: \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\") pod \"praxis-route-af72cf-kserve-8cb896d87-r5qct\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:51.891816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:51.891733 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:43:52.291800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:52.291772 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:43:52.293848 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:43:52.293821 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod645fd7f0_95d0_4208_9383_4e2400ffb7ce.slice/crio-a97e4f9bf867fb43db1ead166c1df8b80669d10b398961f506b57e3f15ec83db WatchSource:0}: Error finding container a97e4f9bf867fb43db1ead166c1df8b80669d10b398961f506b57e3f15ec83db: Status 404 returned error can't find the container with id a97e4f9bf867fb43db1ead166c1df8b80669d10b398961f506b57e3f15ec83db Oct 30 19:43:52.620572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:52.620477 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" event={"ID":"645fd7f0-95d0-4208-9383-4e2400ffb7ce","Type":"ContainerStarted","Data":"4b5b682da96022084e19a95ba84ba9adfb740d39367ce521179552236a14d229"} Oct 30 19:43:52.620572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:52.620536 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" event={"ID":"645fd7f0-95d0-4208-9383-4e2400ffb7ce","Type":"ContainerStarted","Data":"a97e4f9bf867fb43db1ead166c1df8b80669d10b398961f506b57e3f15ec83db"} Oct 30 19:43:52.733355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:52.733305 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" podStartSLOduration=1.733290797 podStartE2EDuration="1.733290797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:52.731982931 +0000 UTC m=+3469.233893416" watchObservedRunningTime="2026-10-30 19:43:52.733290797 +0000 UTC m=+3469.235201282" Oct 30 19:43:52.891971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:43:52.891880 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:44:01.892586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:01.892551 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:44:01.904645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:01.904622 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:44:02.697837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:02.697805 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:44:07.045084 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:07.045044 4789 scope.go:122] "RemoveContainer" containerID="196e0438cf15631c7ce59cc42e8d92f9ab4cc9b98215b9c31df21afc6d4bb0fb" Oct 30 19:44:15.405007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.404975 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:44:15.410803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.410776 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.462362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.462339 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c01685\"/\"kube-root-ca.crt\"" Oct 30 19:44:15.468042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.468015 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c01685\"/\"auth-test-model-c01685-kserve-self-signed-certs\"" Oct 30 19:44:15.498953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.498928 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.499106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.498998 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.499183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.499106 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.499183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.499163 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2d6\" (UniqueName: \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.499406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.499380 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.499481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.499436 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.508899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.508880 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c01685\"/\"openshift-service-ca.crt\"" Oct 30 19:44:15.512137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.512110 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:44:15.562073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.562043 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c01685\"/\"default-dockercfg-54wg5\"" Oct 30 19:44:15.600607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600574 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.600797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600625 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.600797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600752 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.600947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600828 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.600947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600869 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.600947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.600917 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2d6\" (UniqueName: \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.601145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.601121 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.601382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.601360 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.601442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.601381 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.603287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.603265 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.603409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.603292 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.659701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.659632 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2d6\" (UniqueName: \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\") pod \"auth-test-model-c01685-kserve-7dffbdd585-7kxht\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:15.723145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:15.723111 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:16.109012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:16.108987 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:44:16.112156 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:44:16.112131 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod181096a4_aff4_406f_ac5d_f6b6c3d09caf.slice/crio-aa832584de10da723a762a7f411da670e38aeaa6f3a5364c817ccdd477738b4b WatchSource:0}: Error finding container aa832584de10da723a762a7f411da670e38aeaa6f3a5364c817ccdd477738b4b: Status 404 returned error can't find the container with id aa832584de10da723a762a7f411da670e38aeaa6f3a5364c817ccdd477738b4b Oct 30 19:44:16.789991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:16.789959 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" event={"ID":"181096a4-aff4-406f-ac5d-f6b6c3d09caf","Type":"ContainerStarted","Data":"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d"} Oct 30 19:44:16.790440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:16.790000 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" event={"ID":"181096a4-aff4-406f-ac5d-f6b6c3d09caf","Type":"ContainerStarted","Data":"aa832584de10da723a762a7f411da670e38aeaa6f3a5364c817ccdd477738b4b"} Oct 30 19:44:16.903504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:16.903455 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" podStartSLOduration=1.903440796 podStartE2EDuration="1.903440796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:16.901685411 +0000 UTC m=+3493.403595898" watchObservedRunningTime="2026-10-30 19:44:16.903440796 +0000 UTC m=+3493.405351281" Oct 30 19:44:17.723637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:17.723598 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:18.201770 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:44:18.201729 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:44:25.723591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:25.723562 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:25.736344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:25.736311 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:25.865039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:25.865014 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:44:29.387110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:29.387075 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:44:44.408030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.407985 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" podUID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" containerName="main" containerID="cri-o://a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e" gracePeriod=45 Oct 30 19:44:44.661915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.661858 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:44:44.719449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719412 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719472 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719544 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719587 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719630 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\" (UniqueName: \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719760 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\") pod \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\" (UID: \"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8\") " Oct 30 19:44:44.719837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719802 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.719985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.719964 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.720358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.720268 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.720358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.720292 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.720358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.720314 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.721846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.721821 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:44.721846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.721828 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:44:44.721977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.721866 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr" pod "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" (UID: "af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8"). InnerVolumeSpecName "kube-api-access-79dbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:44.821929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.821893 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dbr\" (UniqueName: \"kubernetes.io/projected/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-kube-api-access-79dbr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.821929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.821922 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.821929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.821933 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.822254 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.821942 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:44.987245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.987186 4789 generic.go:356] "Generic (PLEG): container finished" podID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" containerID="a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e" exitCode=0 Oct 30 19:44:44.987447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.987256 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" Oct 30 19:44:44.987447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.987251 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" event={"ID":"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8","Type":"ContainerDied","Data":"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e"} Oct 30 19:44:44.987447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.987371 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj" event={"ID":"af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8","Type":"ContainerDied","Data":"ed545f643936a8ae06d78ffb09846683de8ecb3ea79dc2b61d5fb2f11d33f151"} Oct 30 19:44:44.987447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:44.987394 4789 scope.go:122] "RemoveContainer" containerID="a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e" Oct 30 19:44:45.000499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:45.000475 4789 scope.go:122] "RemoveContainer" containerID="a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e" Oct 30 19:44:45.000761 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:44:45.000741 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e\": container with ID starting with a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e not found: ID does not exist" containerID="a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e" Oct 30 19:44:45.000847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:45.000766 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e"} err="failed to get container status \"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e\": rpc error: code = NotFound desc = could not find container \"a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e\": container with ID starting with a6c358083948c059c6095d00caab95362def4a4e7e1d0a6f499a33ea2741b95e not found: ID does not exist" Oct 30 19:44:45.142833 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:45.142798 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:44:45.147054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:45.147027 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2186d8/auth-test-model-2186d8-kserve-64f6d95c45-qqjmj"] Oct 30 19:44:45.669071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:44:45.669044 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" path="/var/lib/kubelet/pods/af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8/volumes" Oct 30 19:44:48.197953 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:44:48.197908 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:45:00.182724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.182687 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:00.185411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.184182 4789 cpu_manager.go:414] "containerMap: removing container" podUID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" containerName="main" Oct 30 19:45:00.185411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.184226 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" containerName="main" Oct 30 19:45:00.185411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.184444 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4e4e6e-07cd-4a3f-a2ed-f4fe62d02fb8" containerName="main" Oct 30 19:45:00.188504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.188478 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:00.188658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.188617 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.193120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.193096 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:00.193279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.193228 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.197797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.197281 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.228652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.228611 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:00.233993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.233967 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:00.234143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.234060 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.239148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.239122 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:00.239291 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.239270 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.244537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.244516 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.245477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.245457 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:45:00.246065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.246042 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:00.246157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.246117 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:00.250773 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.250752 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.272281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.271773 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:00.274933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.274908 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:00.279176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.279154 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:00.279472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.279445 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.283406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.283385 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.284495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.284474 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:00.288664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.288642 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.292879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.292851 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:00.295784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.295757 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:00.298595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298563 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q69f\" (UniqueName: \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.298699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298654 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrndt\" (UniqueName: \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.298766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298705 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.298766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298755 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrwb\" (UniqueName: \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.298884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298819 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.298884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298848 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmst\" (UniqueName: \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.298996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298916 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.298996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.298991 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.299963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.299929 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:00.304703 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.304678 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:00.313713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.311336 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.326353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.325562 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:00.336412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.336303 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:00.340851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.340828 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:00.346903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.346884 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:00.353464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.353443 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:00.388056 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.388034 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401002 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svshg\" (UniqueName: \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401048 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401093 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401123 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4gg\" (UniqueName: \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401172 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401222 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q69f\" (UniqueName: \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401264 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6qd\" (UniqueName: \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401303 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401337 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401383 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401417 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401485 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrndt\" (UniqueName: \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401544 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401588 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr29m\" (UniqueName: \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401628 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrwb\" (UniqueName: \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401674 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pqxl\" (UniqueName: \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401719 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401747 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401798 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401827 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmst\" (UniqueName: \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401906 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.401931 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnpc\" (UniqueName: \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.402854 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.404089 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.405229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.405019 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.407713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.407693 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:00.407901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.407886 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:00.408730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.408709 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.465729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.465701 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrndt\" (UniqueName: \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\") pod \"maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.469935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.469871 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmst\" (UniqueName: \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.474237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.471278 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q69f\" (UniqueName: \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\") pod \"maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.474237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.473901 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrwb\" (UniqueName: \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\") pod \"maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.503532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503505 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4gg\" (UniqueName: \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.503671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503571 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.503671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503622 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6qd\" (UniqueName: \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.503671 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503651 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.503870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503685 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.503870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503726 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.503870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503767 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.503870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503849 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr29m\" (UniqueName: \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.504124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503903 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pqxl\" (UniqueName: \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.504124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503943 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.504124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.503969 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.504124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504075 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnpc\" (UniqueName: \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.504385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504143 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svshg\" (UniqueName: \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.504385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504178 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.504385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504272 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504704 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504834 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.504959 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.505190 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.505566 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.505620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.505573 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.506289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.506271 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.513288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.513265 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:00.521515 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.521352 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:00.550006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.549976 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:00.578999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.578558 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6qd\" (UniqueName: \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.582872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.582357 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pqxl\" (UniqueName: \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.586304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.585853 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnpc\" (UniqueName: \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.590010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.589635 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:00.590964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.590929 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr29m\" (UniqueName: \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:00.593298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.592192 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\") pod \"maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.593298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.592226 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4gg\" (UniqueName: \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\") pod \"maas-api-key-cleanup-29889825-wpm77\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.594401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.594087 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svshg\" (UniqueName: \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\") pod \"maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.600225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.597898 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:00.620140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.619908 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:00.632469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.629974 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:00.649987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.644194 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:00.870981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.866774 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:00.879414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:00.879162 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:01.219518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.219488 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:01.266250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.266201 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:01.268770 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:01.268713 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7ca63d_d889_4bcc_bbcc_296d72c836b2.slice/crio-e1bc8e5b98ff94e0cfeeb04c9157d516f97c1f73df2deaa95face8170afd8b15 WatchSource:0}: Error finding container e1bc8e5b98ff94e0cfeeb04c9157d516f97c1f73df2deaa95face8170afd8b15: Status 404 returned error can't find the container with id e1bc8e5b98ff94e0cfeeb04c9157d516f97c1f73df2deaa95face8170afd8b15 Oct 30 19:45:01.270702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.270675 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:01.272825 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:01.272802 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8e6189_8748_4260_b2c2_51f2f07f11b4.slice/crio-8972852ff08c6ed691387235f3acab58a41c8f38787476a2fe2b36c9f1441b81 WatchSource:0}: Error finding container 8972852ff08c6ed691387235f3acab58a41c8f38787476a2fe2b36c9f1441b81: Status 404 returned error can't find the container with id 8972852ff08c6ed691387235f3acab58a41c8f38787476a2fe2b36c9f1441b81 Oct 30 19:45:01.315522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.315493 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:01.321145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.321121 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:01.324308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.324289 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:01.357664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.357638 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:01.408467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.400257 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:01.429405 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:01.429371 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d893c93_da6c_44eb_8d06_81fd692e969e.slice/crio-720fe2f22cb6d58da78f51b3a83aabf98ffb778d15f169176f47d794407a9a8f WatchSource:0}: Error finding container 720fe2f22cb6d58da78f51b3a83aabf98ffb778d15f169176f47d794407a9a8f: Status 404 returned error can't find the container with id 720fe2f22cb6d58da78f51b3a83aabf98ffb778d15f169176f47d794407a9a8f Oct 30 19:45:01.455655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.455630 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:01.555484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.555421 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:01.558561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:01.558534 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:01.596672 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:01.596603 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf115a03b_628e_45a6_9e20_004f39e3b63a.slice/crio-c826287be475cdc725780500deaf49d9e6687226be46394f7ce8eb990e006d54 WatchSource:0}: Error finding container c826287be475cdc725780500deaf49d9e6687226be46394f7ce8eb990e006d54: Status 404 returned error can't find the container with id c826287be475cdc725780500deaf49d9e6687226be46394f7ce8eb990e006d54 Oct 30 19:45:01.597800 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:01.597764 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ccb14e_6b93_41f5_bb36_df6ee7317e89.slice/crio-d2c00939963856fc3d25e2283eaf4e797386ff29c713a138e36370c7784552cc WatchSource:0}: Error finding container d2c00939963856fc3d25e2283eaf4e797386ff29c713a138e36370c7784552cc: Status 404 returned error can't find the container with id d2c00939963856fc3d25e2283eaf4e797386ff29c713a138e36370c7784552cc Oct 30 19:45:02.109537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.109453 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerStarted","Data":"d2c00939963856fc3d25e2283eaf4e797386ff29c713a138e36370c7784552cc"} Oct 30 19:45:02.111578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.111547 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerStarted","Data":"e1237574d6c12da96cb412bed5eb228a64fe0171e24dd495b12e68e07a803255"} Oct 30 19:45:02.113077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.113053 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerStarted","Data":"3caaa293b216929c5437467db7793edc83a797e4bf8652a77e0508746c6da47a"} Oct 30 19:45:02.113175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.113101 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerStarted","Data":"8972852ff08c6ed691387235f3acab58a41c8f38787476a2fe2b36c9f1441b81"} Oct 30 19:45:02.114376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.114349 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerStarted","Data":"0e4af09f2509306157ff24d2db0099fe48c64dc9a1c9440457a12eb9179d27e3"} Oct 30 19:45:02.116141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.116117 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerStarted","Data":"6e037a05b8ac05641f017a64989c1acfbd41d5ca0733549ce04a74dbe0e27a1a"} Oct 30 19:45:02.116250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.116161 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerStarted","Data":"981edec9a2155bacc44fc9ec5ae386ba153075ea9b8d04a6fecee321789b0a6f"} Oct 30 19:45:02.117376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.117354 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerStarted","Data":"c826287be475cdc725780500deaf49d9e6687226be46394f7ce8eb990e006d54"} Oct 30 19:45:02.118801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.118778 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerStarted","Data":"84f565ed0e8e515fa8c61917fc224202e0ccc24815c350f68596b3f0636bc1e4"} Oct 30 19:45:02.118894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.118820 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerStarted","Data":"e1bc8e5b98ff94e0cfeeb04c9157d516f97c1f73df2deaa95face8170afd8b15"} Oct 30 19:45:02.119983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.119959 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerStarted","Data":"ef7045c774054dd15a974ccc1eb1b18625e1eff52af0c1454f099312a22dd498"} Oct 30 19:45:02.121391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.121366 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerStarted","Data":"aec1b85376d79c24084812705d1ae65f43b8917c363357616418bfefd2b366c2"} Oct 30 19:45:02.121472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.121410 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerStarted","Data":"77e3812cdd9ce84fc681194c1be2298d2304507c212594bef4cfa129791d41ce"} Oct 30 19:45:02.122493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.122473 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerStarted","Data":"720fe2f22cb6d58da78f51b3a83aabf98ffb778d15f169176f47d794407a9a8f"} Oct 30 19:45:02.123500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.123479 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerStarted","Data":"e35c90aa57fb3a055075fc42b7d5879df5592cb1638af3bb51de8a75281b5661"} Oct 30 19:45:02.226139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.226082 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" podStartSLOduration=2.226063494 podStartE2EDuration="2.226063494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.224226201 +0000 UTC m=+3538.726136688" watchObservedRunningTime="2026-10-30 19:45:02.226063494 +0000 UTC m=+3538.727973981" Oct 30 19:45:02.338323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.338271 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" podStartSLOduration=2.338258935 podStartE2EDuration="2.338258935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.335389649 +0000 UTC m=+3538.837300138" watchObservedRunningTime="2026-10-30 19:45:02.338258935 +0000 UTC m=+3538.840169418" Oct 30 19:45:02.454085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.454035 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" podStartSLOduration=2.454020422 podStartE2EDuration="2.454020422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.447248441 +0000 UTC m=+3538.949158929" watchObservedRunningTime="2026-10-30 19:45:02.454020422 +0000 UTC m=+3538.955930910" Oct 30 19:45:02.560550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:02.560504 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" podStartSLOduration=2.560491072 podStartE2EDuration="2.560491072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.558754457 +0000 UTC m=+3539.060664944" watchObservedRunningTime="2026-10-30 19:45:02.560491072 +0000 UTC m=+3539.062401558" Oct 30 19:45:03.134574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.134536 4789 generic.go:356] "Generic (PLEG): container finished" podID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerID="e6a5fdbd737b38d341450d538b8d6ca8a972923854ded6203f4f8520b9a5b687" exitCode=60 Oct 30 19:45:03.134743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.134631 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerDied","Data":"e6a5fdbd737b38d341450d538b8d6ca8a972923854ded6203f4f8520b9a5b687"} Oct 30 19:45:03.134959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.134937 4789 scope.go:122] "RemoveContainer" containerID="e6a5fdbd737b38d341450d538b8d6ca8a972923854ded6203f4f8520b9a5b687" Oct 30 19:45:03.136911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.136887 4789 generic.go:356] "Generic (PLEG): container finished" podID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerID="518c115da605b2913e916fcfe563609997d54ca46256060af862b0699243a9bd" exitCode=60 Oct 30 19:45:03.137002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.136913 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerDied","Data":"518c115da605b2913e916fcfe563609997d54ca46256060af862b0699243a9bd"} Oct 30 19:45:03.137451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.137431 4789 scope.go:122] "RemoveContainer" containerID="518c115da605b2913e916fcfe563609997d54ca46256060af862b0699243a9bd" Oct 30 19:45:03.143995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.143490 4789 generic.go:356] "Generic (PLEG): container finished" podID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerID="a406869dd917c6ad692f3e7e21683682256b20af9493c66ceae61d85b5d70667" exitCode=60 Oct 30 19:45:03.143995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.143609 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerDied","Data":"a406869dd917c6ad692f3e7e21683682256b20af9493c66ceae61d85b5d70667"} Oct 30 19:45:03.143995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.143809 4789 scope.go:122] "RemoveContainer" containerID="a406869dd917c6ad692f3e7e21683682256b20af9493c66ceae61d85b5d70667" Oct 30 19:45:03.145518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.145495 4789 generic.go:356] "Generic (PLEG): container finished" podID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerID="b1bb49f850330949df519953bab52ae8d03bea46fcd955d436b9a388c94b80e9" exitCode=60 Oct 30 19:45:03.145622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.145578 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerDied","Data":"b1bb49f850330949df519953bab52ae8d03bea46fcd955d436b9a388c94b80e9"} Oct 30 19:45:03.145806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.145788 4789 scope.go:122] "RemoveContainer" containerID="b1bb49f850330949df519953bab52ae8d03bea46fcd955d436b9a388c94b80e9" Oct 30 19:45:03.147921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.147895 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerID="3caaa293b216929c5437467db7793edc83a797e4bf8652a77e0508746c6da47a" exitCode=60 Oct 30 19:45:03.148053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.147997 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerDied","Data":"3caaa293b216929c5437467db7793edc83a797e4bf8652a77e0508746c6da47a"} Oct 30 19:45:03.148256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.148190 4789 scope.go:122] "RemoveContainer" containerID="3caaa293b216929c5437467db7793edc83a797e4bf8652a77e0508746c6da47a" Oct 30 19:45:03.154290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.154019 4789 generic.go:356] "Generic (PLEG): container finished" podID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerID="1e7b1570b261ea8f8810fa4f2c5f94f3768c122cdb15848b9bc624f4d1ea8b34" exitCode=60 Oct 30 19:45:03.154290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.154093 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerDied","Data":"1e7b1570b261ea8f8810fa4f2c5f94f3768c122cdb15848b9bc624f4d1ea8b34"} Oct 30 19:45:03.156171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.154408 4789 scope.go:122] "RemoveContainer" containerID="1e7b1570b261ea8f8810fa4f2c5f94f3768c122cdb15848b9bc624f4d1ea8b34" Oct 30 19:45:03.158844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.158820 4789 generic.go:356] "Generic (PLEG): container finished" podID="934493f5-bc4a-4863-8273-824889175a5a" containerID="6e037a05b8ac05641f017a64989c1acfbd41d5ca0733549ce04a74dbe0e27a1a" exitCode=60 Oct 30 19:45:03.158946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.158894 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerDied","Data":"6e037a05b8ac05641f017a64989c1acfbd41d5ca0733549ce04a74dbe0e27a1a"} Oct 30 19:45:03.159273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.159253 4789 scope.go:122] "RemoveContainer" containerID="6e037a05b8ac05641f017a64989c1acfbd41d5ca0733549ce04a74dbe0e27a1a" Oct 30 19:45:03.161637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.161576 4789 generic.go:356] "Generic (PLEG): container finished" podID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerID="8d8a76d7c766072217e75186d37fc1e168152bbff32da099027bbb6903ac72a8" exitCode=60 Oct 30 19:45:03.161721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.161654 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerDied","Data":"8d8a76d7c766072217e75186d37fc1e168152bbff32da099027bbb6903ac72a8"} Oct 30 19:45:03.161834 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.161818 4789 scope.go:122] "RemoveContainer" containerID="8d8a76d7c766072217e75186d37fc1e168152bbff32da099027bbb6903ac72a8" Oct 30 19:45:03.163507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.163484 4789 generic.go:356] "Generic (PLEG): container finished" podID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerID="84f565ed0e8e515fa8c61917fc224202e0ccc24815c350f68596b3f0636bc1e4" exitCode=60 Oct 30 19:45:03.163626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.163569 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerDied","Data":"84f565ed0e8e515fa8c61917fc224202e0ccc24815c350f68596b3f0636bc1e4"} Oct 30 19:45:03.163777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.163757 4789 scope.go:122] "RemoveContainer" containerID="84f565ed0e8e515fa8c61917fc224202e0ccc24815c350f68596b3f0636bc1e4" Oct 30 19:45:03.165332 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.165312 4789 generic.go:356] "Generic (PLEG): container finished" podID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerID="edc1ff0e13b29ac6f991a9494986191291f6d6a48db6701dec4c7f9d33aeeee6" exitCode=60 Oct 30 19:45:03.165419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.165334 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerDied","Data":"edc1ff0e13b29ac6f991a9494986191291f6d6a48db6701dec4c7f9d33aeeee6"} Oct 30 19:45:03.165594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.165578 4789 scope.go:122] "RemoveContainer" containerID="edc1ff0e13b29ac6f991a9494986191291f6d6a48db6701dec4c7f9d33aeeee6" Oct 30 19:45:03.167019 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.166991 4789 generic.go:356] "Generic (PLEG): container finished" podID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerID="aec1b85376d79c24084812705d1ae65f43b8917c363357616418bfefd2b366c2" exitCode=60 Oct 30 19:45:03.167099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.167015 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerDied","Data":"aec1b85376d79c24084812705d1ae65f43b8917c363357616418bfefd2b366c2"} Oct 30 19:45:03.167297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.167278 4789 scope.go:122] "RemoveContainer" containerID="aec1b85376d79c24084812705d1ae65f43b8917c363357616418bfefd2b366c2" Oct 30 19:45:03.356071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.356033 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:45:03.362631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.362610 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.429529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.429494 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-59594f\"/\"kube-root-ca.crt\"" Oct 30 19:45:03.430475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.430448 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-59594f\"/\"openshift-service-ca.crt\"" Oct 30 19:45:03.431047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.431025 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-59594f\"/\"default-dockercfg-nmxn2\"" Oct 30 19:45:03.431248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.429494 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-59594f\"/\"auth-test-model-59594f-kserve-self-signed-certs\"" Oct 30 19:45:03.441406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.441374 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:45:03.464931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.464894 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.465126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.464939 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.465126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.465093 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.465277 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.465163 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.465342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.465302 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.465415 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.465344 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4svz\" (UniqueName: \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566410 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566447 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566491 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566531 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566931 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.566959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566930 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.567173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.566969 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.567173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.567011 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4svz\" (UniqueName: \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.567331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.567198 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.569083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.569055 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.569236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.569200 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.627263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.627200 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4svz\" (UniqueName: \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\") pod \"auth-test-model-59594f-kserve-7b697ff5b5-pfvsv\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:03.676227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:03.675995 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:04.123643 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.123621 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:45:04.126482 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:45:04.126441 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a4374f_abb0_41de_9a72_0befba6078b7.slice/crio-2bd12e49b0e38206b8fbf8ede6d3f55be5e2dca8870c7be73e67b25ccd7bcebe WatchSource:0}: Error finding container 2bd12e49b0e38206b8fbf8ede6d3f55be5e2dca8870c7be73e67b25ccd7bcebe: Status 404 returned error can't find the container with id 2bd12e49b0e38206b8fbf8ede6d3f55be5e2dca8870c7be73e67b25ccd7bcebe Oct 30 19:45:04.177014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.176979 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerStarted","Data":"eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0"} Oct 30 19:45:04.179184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.179158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerStarted","Data":"c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c"} Oct 30 19:45:04.181238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.181185 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerStarted","Data":"0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9"} Oct 30 19:45:04.183360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.183340 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerStarted","Data":"6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463"} Oct 30 19:45:04.184571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.184550 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" event={"ID":"75a4374f-abb0-41de-9a72-0befba6078b7","Type":"ContainerStarted","Data":"2bd12e49b0e38206b8fbf8ede6d3f55be5e2dca8870c7be73e67b25ccd7bcebe"} Oct 30 19:45:04.186362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.186310 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerStarted","Data":"c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4"} Oct 30 19:45:04.188986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.188961 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerStarted","Data":"60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333"} Oct 30 19:45:04.201338 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.201310 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerStarted","Data":"a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603"} Oct 30 19:45:04.204346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.204316 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerStarted","Data":"de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a"} Oct 30 19:45:04.206679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.206642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerStarted","Data":"5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df"} Oct 30 19:45:04.208750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.208726 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerStarted","Data":"250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f"} Oct 30 19:45:04.210502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.210481 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerStarted","Data":"011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba"} Oct 30 19:45:04.687097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:04.687039 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" podStartSLOduration=4.687025692 podStartE2EDuration="4.687025692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:04.544447459 +0000 UTC m=+3541.046357955" watchObservedRunningTime="2026-10-30 19:45:04.687025692 +0000 UTC m=+3541.188936177" Oct 30 19:45:05.147077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.146977 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" podStartSLOduration=5.146963887 podStartE2EDuration="5.146963887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.144569371 +0000 UTC m=+3541.646479858" watchObservedRunningTime="2026-10-30 19:45:05.146963887 +0000 UTC m=+3541.648874373" Oct 30 19:45:05.219563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.219533 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerID="c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c" exitCode=60 Oct 30 19:45:05.219793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.219567 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerDied","Data":"c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c"} Oct 30 19:45:05.219793 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.219627 4789 scope.go:122] "RemoveContainer" containerID="3caaa293b216929c5437467db7793edc83a797e4bf8652a77e0508746c6da47a" Oct 30 19:45:05.219940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.219917 4789 scope.go:122] "RemoveContainer" containerID="c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c" Oct 30 19:45:05.220305 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.220267 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8_odh-ai-gateway-infra(ef8e6189-8748-4260-b2c2-51f2f07f11b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" Oct 30 19:45:05.221828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.221796 4789 generic.go:356] "Generic (PLEG): container finished" podID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerID="0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9" exitCode=60 Oct 30 19:45:05.221935 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.221861 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerDied","Data":"0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9"} Oct 30 19:45:05.222133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.222112 4789 scope.go:122] "RemoveContainer" containerID="0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9" Oct 30 19:45:05.222458 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.222435 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s_odh-ai-gateway-infra(a6eb9bc0-34ac-47f6-97c8-c9734451f08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" Oct 30 19:45:05.223683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.223665 4789 generic.go:356] "Generic (PLEG): container finished" podID="934493f5-bc4a-4863-8273-824889175a5a" containerID="6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463" exitCode=60 Oct 30 19:45:05.223797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.223726 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerDied","Data":"6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463"} Oct 30 19:45:05.223989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.223972 4789 scope.go:122] "RemoveContainer" containerID="6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463" Oct 30 19:45:05.224406 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.224225 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh_odh-ai-gateway-infra(934493f5-bc4a-4863-8273-824889175a5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" podUID="934493f5-bc4a-4863-8273-824889175a5a" Oct 30 19:45:05.225431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.225405 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" event={"ID":"75a4374f-abb0-41de-9a72-0befba6078b7","Type":"ContainerStarted","Data":"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd"} Oct 30 19:45:05.227120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.227094 4789 generic.go:356] "Generic (PLEG): container finished" podID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerID="c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4" exitCode=60 Oct 30 19:45:05.227229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.227179 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerDied","Data":"c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4"} Oct 30 19:45:05.227406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.227388 4789 scope.go:122] "RemoveContainer" containerID="c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4" Oct 30 19:45:05.227624 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.227606 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd_odh-ai-gateway-infra(f115a03b-628e-45a6-9e20-004f39e3b63a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" Oct 30 19:45:05.230239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.230200 4789 generic.go:356] "Generic (PLEG): container finished" podID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerID="60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333" exitCode=60 Oct 30 19:45:05.230340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.230309 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerDied","Data":"60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333"} Oct 30 19:45:05.230495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.230477 4789 scope.go:122] "RemoveContainer" containerID="60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333" Oct 30 19:45:05.230698 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.230678 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7_odh-ai-gateway-infra(1c7ca63d-d889-4bcc-bbcc-296d72c836b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" Oct 30 19:45:05.232276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.232253 4789 generic.go:356] "Generic (PLEG): container finished" podID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerID="a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603" exitCode=60 Oct 30 19:45:05.232385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.232276 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerDied","Data":"a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603"} Oct 30 19:45:05.232530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.232512 4789 scope.go:122] "RemoveContainer" containerID="a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603" Oct 30 19:45:05.232756 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.232736 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-wpm77_odh-ai-gateway-infra(fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" Oct 30 19:45:05.234305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.234284 4789 generic.go:356] "Generic (PLEG): container finished" podID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerID="de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a" exitCode=60 Oct 30 19:45:05.234406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.234304 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerDied","Data":"de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a"} Oct 30 19:45:05.234567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.234523 4789 scope.go:122] "RemoveContainer" containerID="de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a" Oct 30 19:45:05.234715 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.234689 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx_odh-ai-gateway-infra(c8a1e201-d495-43ed-bbf8-e2924186f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" Oct 30 19:45:05.234715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.234707 4789 scope.go:122] "RemoveContainer" containerID="1e7b1570b261ea8f8810fa4f2c5f94f3768c122cdb15848b9bc624f4d1ea8b34" Oct 30 19:45:05.236374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.236356 4789 generic.go:356] "Generic (PLEG): container finished" podID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerID="5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df" exitCode=60 Oct 30 19:45:05.236455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.236423 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerDied","Data":"5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df"} Oct 30 19:45:05.236714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.236675 4789 scope.go:122] "RemoveContainer" containerID="5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df" Oct 30 19:45:05.236909 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.236889 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr_odh-ai-gateway-infra(6d893c93-da6c-44eb-8d06-81fd692e969e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" Oct 30 19:45:05.238129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.238111 4789 generic.go:356] "Generic (PLEG): container finished" podID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerID="250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f" exitCode=60 Oct 30 19:45:05.238222 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.238172 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerDied","Data":"250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f"} Oct 30 19:45:05.238414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.238395 4789 scope.go:122] "RemoveContainer" containerID="250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f" Oct 30 19:45:05.238612 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.238590 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5_odh-ai-gateway-infra(1fd465a2-3468-4755-af32-d0d22bbc6036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" Oct 30 19:45:05.240179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.240158 4789 generic.go:356] "Generic (PLEG): container finished" podID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerID="011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba" exitCode=60 Oct 30 19:45:05.240278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.240178 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerDied","Data":"011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba"} Oct 30 19:45:05.240453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.240435 4789 scope.go:122] "RemoveContainer" containerID="011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba" Oct 30 19:45:05.240712 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.240686 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9_odh-ai-gateway-infra(00ccb14e-6b93-41f5-bb36-df6ee7317e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" Oct 30 19:45:05.242146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.242125 4789 generic.go:356] "Generic (PLEG): container finished" podID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerID="eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0" exitCode=60 Oct 30 19:45:05.242251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.242176 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerDied","Data":"eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0"} Oct 30 19:45:05.242377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.242362 4789 scope.go:122] "RemoveContainer" containerID="eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0" Oct 30 19:45:05.242548 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:05.242530 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp_odh-ai-gateway-infra(a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" Oct 30 19:45:05.247863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.247844 4789 scope.go:122] "RemoveContainer" containerID="6e037a05b8ac05641f017a64989c1acfbd41d5ca0733549ce04a74dbe0e27a1a" Oct 30 19:45:05.260629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.260574 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" podStartSLOduration=5.260556498 podStartE2EDuration="5.260556498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.254426166 +0000 UTC m=+3541.756336652" watchObservedRunningTime="2026-10-30 19:45:05.260556498 +0000 UTC m=+3541.762466985" Oct 30 19:45:05.261639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.261624 4789 scope.go:122] "RemoveContainer" containerID="8d8a76d7c766072217e75186d37fc1e168152bbff32da099027bbb6903ac72a8" Oct 30 19:45:05.274174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.274159 4789 scope.go:122] "RemoveContainer" containerID="84f565ed0e8e515fa8c61917fc224202e0ccc24815c350f68596b3f0636bc1e4" Oct 30 19:45:05.290360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.290338 4789 scope.go:122] "RemoveContainer" containerID="edc1ff0e13b29ac6f991a9494986191291f6d6a48db6701dec4c7f9d33aeeee6" Oct 30 19:45:05.302952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.302935 4789 scope.go:122] "RemoveContainer" containerID="aec1b85376d79c24084812705d1ae65f43b8917c363357616418bfefd2b366c2" Oct 30 19:45:05.316596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.316577 4789 scope.go:122] "RemoveContainer" containerID="e6a5fdbd737b38d341450d538b8d6ca8a972923854ded6203f4f8520b9a5b687" Oct 30 19:45:05.329350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.329330 4789 scope.go:122] "RemoveContainer" containerID="518c115da605b2913e916fcfe563609997d54ca46256060af862b0699243a9bd" Oct 30 19:45:05.341420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.341394 4789 scope.go:122] "RemoveContainer" containerID="a406869dd917c6ad692f3e7e21683682256b20af9493c66ceae61d85b5d70667" Oct 30 19:45:05.353721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.353700 4789 scope.go:122] "RemoveContainer" containerID="b1bb49f850330949df519953bab52ae8d03bea46fcd955d436b9a388c94b80e9" Oct 30 19:45:05.365386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.365346 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" podStartSLOduration=5.365335254 podStartE2EDuration="5.365335254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.364131732 +0000 UTC m=+3541.866042227" watchObservedRunningTime="2026-10-30 19:45:05.365335254 +0000 UTC m=+3541.867245740" Oct 30 19:45:05.483873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.483826 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" podStartSLOduration=5.483809932 podStartE2EDuration="5.483809932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.483019187 +0000 UTC m=+3541.984929672" watchObservedRunningTime="2026-10-30 19:45:05.483809932 +0000 UTC m=+3541.985720417" Oct 30 19:45:05.596364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.596312 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" podStartSLOduration=5.596299403 podStartE2EDuration="5.596299403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.593888485 +0000 UTC m=+3542.095798971" watchObservedRunningTime="2026-10-30 19:45:05.596299403 +0000 UTC m=+3542.098209889" Oct 30 19:45:05.676711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.676683 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:05.820072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:05.819973 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" podStartSLOduration=5.819959825 podStartE2EDuration="5.819959825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.819015256 +0000 UTC m=+3542.320925742" watchObservedRunningTime="2026-10-30 19:45:05.819959825 +0000 UTC m=+3542.321870311" Oct 30 19:45:06.251849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.251824 4789 scope.go:122] "RemoveContainer" containerID="de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a" Oct 30 19:45:06.252254 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.252202 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx_odh-ai-gateway-infra(c8a1e201-d495-43ed-bbf8-e2924186f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" Oct 30 19:45:06.253591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.253569 4789 scope.go:122] "RemoveContainer" containerID="5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df" Oct 30 19:45:06.253839 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.253821 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr_odh-ai-gateway-infra(6d893c93-da6c-44eb-8d06-81fd692e969e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" Oct 30 19:45:06.255053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.255031 4789 scope.go:122] "RemoveContainer" containerID="250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f" Oct 30 19:45:06.255306 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.255287 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5_odh-ai-gateway-infra(1fd465a2-3468-4755-af32-d0d22bbc6036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" Oct 30 19:45:06.256461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.256443 4789 scope.go:122] "RemoveContainer" containerID="011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba" Oct 30 19:45:06.256678 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.256658 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9_odh-ai-gateway-infra(00ccb14e-6b93-41f5-bb36-df6ee7317e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" Oct 30 19:45:06.257971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.257946 4789 scope.go:122] "RemoveContainer" containerID="eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0" Oct 30 19:45:06.258172 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.258153 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp_odh-ai-gateway-infra(a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" Oct 30 19:45:06.259398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.259380 4789 scope.go:122] "RemoveContainer" containerID="c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c" Oct 30 19:45:06.259599 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.259579 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8_odh-ai-gateway-infra(ef8e6189-8748-4260-b2c2-51f2f07f11b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" Oct 30 19:45:06.260847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.260829 4789 scope.go:122] "RemoveContainer" containerID="0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9" Oct 30 19:45:06.261044 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.261023 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s_odh-ai-gateway-infra(a6eb9bc0-34ac-47f6-97c8-c9734451f08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" Oct 30 19:45:06.262453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.262436 4789 scope.go:122] "RemoveContainer" containerID="6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463" Oct 30 19:45:06.262624 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.262604 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh_odh-ai-gateway-infra(934493f5-bc4a-4863-8273-824889175a5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" podUID="934493f5-bc4a-4863-8273-824889175a5a" Oct 30 19:45:06.263906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.263888 4789 scope.go:122] "RemoveContainer" containerID="c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4" Oct 30 19:45:06.264088 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.264068 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd_odh-ai-gateway-infra(f115a03b-628e-45a6-9e20-004f39e3b63a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" Oct 30 19:45:06.265293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.265275 4789 scope.go:122] "RemoveContainer" containerID="60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333" Oct 30 19:45:06.265468 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.265449 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7_odh-ai-gateway-infra(1c7ca63d-d889-4bcc-bbcc-296d72c836b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" Oct 30 19:45:06.266826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.266805 4789 scope.go:122] "RemoveContainer" containerID="a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603" Oct 30 19:45:06.267042 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:06.267019 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-wpm77_odh-ai-gateway-infra(fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" Oct 30 19:45:06.779033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:06.778981 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" podStartSLOduration=3.778968776 podStartE2EDuration="3.778968776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:06.776123954 +0000 UTC m=+3543.278034440" watchObservedRunningTime="2026-10-30 19:45:06.778968776 +0000 UTC m=+3543.280879262" Oct 30 19:45:10.731858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:10.731816 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:45:10.732383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:10.732064 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" containerID="cri-o://527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26" gracePeriod=30 Oct 30 19:45:13.677188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:13.677111 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:13.689841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:13.689818 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:14.343553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:14.343519 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:45:15.487175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.487140 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:15.487555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.487188 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:15.661708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.661668 4789 scope.go:122] "RemoveContainer" containerID="c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c" Oct 30 19:45:15.661907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.661876 4789 scope.go:122] "RemoveContainer" containerID="250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f" Oct 30 19:45:15.661994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.661976 4789 scope.go:122] "RemoveContainer" containerID="011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba" Oct 30 19:45:15.662197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:15.662154 4789 scope.go:122] "RemoveContainer" containerID="0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9" Oct 30 19:45:16.129753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.129732 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:45:16.229288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229258 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229318 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229356 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229385 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229428 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229502 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229530 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229539 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.229615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229571 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229722 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\") pod \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\" (UID: \"6a18a7cf-efc4-451c-9d94-7d6213eaec43\") " Oct 30 19:45:16.229853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229774 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.229911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.229876 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230006 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230037 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230288 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230308 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230323 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230336 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.230374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.230351 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.231901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.231872 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:16.232006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.231986 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.232085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.232068 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:16.232248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.232202 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w" pod "6a18a7cf-efc4-451c-9d94-7d6213eaec43" (UID: "6a18a7cf-efc4-451c-9d94-7d6213eaec43"). InnerVolumeSpecName "kube-api-access-6jf5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:16.331601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.331525 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.331601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.331554 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jf5w\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-kube-api-access-6jf5w\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.331601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.331565 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.331601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.331576 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a18a7cf-efc4-451c-9d94-7d6213eaec43-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:16.344267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.344199 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerStarted","Data":"6f362e4648b5f7281b92eb660a27c5d1a13a46fe1a547a62753f0d5d3ea325d5"} Oct 30 19:45:16.346178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.346155 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerStarted","Data":"e872320d9ba7998e36325f4dc6efb94bfd071864d6e10aa7e543d09a74491a3e"} Oct 30 19:45:16.348033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.348011 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerStarted","Data":"077cb8f086521370869c8d08d7125791232668a0c9702a72889a364927cd2a3f"} Oct 30 19:45:16.349985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.349963 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerStarted","Data":"3b0c3242d4277967394fed02f7257199a1acf4b6d0dfeb8b4226ee8959c6f2cb"} Oct 30 19:45:16.351267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.351245 4789 generic.go:356] "Generic (PLEG): container finished" podID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerID="527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26" exitCode=0 Oct 30 19:45:16.351377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.351277 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" Oct 30 19:45:16.351377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.351324 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" event={"ID":"6a18a7cf-efc4-451c-9d94-7d6213eaec43","Type":"ContainerDied","Data":"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26"} Oct 30 19:45:16.351377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.351373 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv" event={"ID":"6a18a7cf-efc4-451c-9d94-7d6213eaec43","Type":"ContainerDied","Data":"3f204607a6748ac7bc55b3b12710b1eaba2e88f266d2a73a6bedfd0020dedb30"} Oct 30 19:45:16.351572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.351402 4789 scope.go:122] "RemoveContainer" containerID="527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26" Oct 30 19:45:16.361761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.361741 4789 scope.go:122] "RemoveContainer" containerID="527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26" Oct 30 19:45:16.362012 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:16.361994 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26\": container with ID starting with 527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26 not found: ID does not exist" containerID="527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26" Oct 30 19:45:16.362058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.362019 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26"} err="failed to get container status \"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26\": rpc error: code = NotFound desc = could not find container \"527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26\": container with ID starting with 527d02f35d529e23240612e2caa4b227b7395c28ad645f2ce4119f51d657eb26 not found: ID does not exist" Oct 30 19:45:16.660946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.660854 4789 scope.go:122] "RemoveContainer" containerID="c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4" Oct 30 19:45:16.661421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.660980 4789 scope.go:122] "RemoveContainer" containerID="60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333" Oct 30 19:45:16.661421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.661037 4789 scope.go:122] "RemoveContainer" containerID="a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603" Oct 30 19:45:16.661421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.661143 4789 scope.go:122] "RemoveContainer" containerID="de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a" Oct 30 19:45:16.799410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.799194 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:45:16.809602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:16.809414 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-twggv"] Oct 30 19:45:17.361849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.361818 4789 generic.go:356] "Generic (PLEG): container finished" podID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerID="e872320d9ba7998e36325f4dc6efb94bfd071864d6e10aa7e543d09a74491a3e" exitCode=60 Oct 30 19:45:17.362029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.361884 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerDied","Data":"e872320d9ba7998e36325f4dc6efb94bfd071864d6e10aa7e543d09a74491a3e"} Oct 30 19:45:17.362029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.361953 4789 scope.go:122] "RemoveContainer" containerID="011b88b9a56cbbdb0992cab8a46a0679fd42b00c6bbd10eb1b19683e04b1d9ba" Oct 30 19:45:17.362367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.362351 4789 scope.go:122] "RemoveContainer" containerID="e872320d9ba7998e36325f4dc6efb94bfd071864d6e10aa7e543d09a74491a3e" Oct 30 19:45:17.362717 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:17.362693 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9_odh-ai-gateway-infra(00ccb14e-6b93-41f5-bb36-df6ee7317e89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" Oct 30 19:45:17.364560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.364532 4789 generic.go:356] "Generic (PLEG): container finished" podID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerID="077cb8f086521370869c8d08d7125791232668a0c9702a72889a364927cd2a3f" exitCode=60 Oct 30 19:45:17.364670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.364607 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerDied","Data":"077cb8f086521370869c8d08d7125791232668a0c9702a72889a364927cd2a3f"} Oct 30 19:45:17.365016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.364995 4789 scope.go:122] "RemoveContainer" containerID="077cb8f086521370869c8d08d7125791232668a0c9702a72889a364927cd2a3f" Oct 30 19:45:17.365304 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:17.365285 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8_odh-ai-gateway-infra(ef8e6189-8748-4260-b2c2-51f2f07f11b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" Oct 30 19:45:17.366660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.366642 4789 generic.go:356] "Generic (PLEG): container finished" podID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerID="3b0c3242d4277967394fed02f7257199a1acf4b6d0dfeb8b4226ee8959c6f2cb" exitCode=60 Oct 30 19:45:17.366754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.366713 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerDied","Data":"3b0c3242d4277967394fed02f7257199a1acf4b6d0dfeb8b4226ee8959c6f2cb"} Oct 30 19:45:17.367033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.367017 4789 scope.go:122] "RemoveContainer" containerID="3b0c3242d4277967394fed02f7257199a1acf4b6d0dfeb8b4226ee8959c6f2cb" Oct 30 19:45:17.367310 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:17.367284 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s_odh-ai-gateway-infra(a6eb9bc0-34ac-47f6-97c8-c9734451f08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" Oct 30 19:45:17.369778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.369761 4789 generic.go:356] "Generic (PLEG): container finished" podID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerID="6f362e4648b5f7281b92eb660a27c5d1a13a46fe1a547a62753f0d5d3ea325d5" exitCode=60 Oct 30 19:45:17.369867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.369790 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerDied","Data":"6f362e4648b5f7281b92eb660a27c5d1a13a46fe1a547a62753f0d5d3ea325d5"} Oct 30 19:45:17.370079 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.370063 4789 scope.go:122] "RemoveContainer" containerID="6f362e4648b5f7281b92eb660a27c5d1a13a46fe1a547a62753f0d5d3ea325d5" Oct 30 19:45:17.370292 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:17.370275 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5_odh-ai-gateway-infra(1fd465a2-3468-4755-af32-d0d22bbc6036)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" Oct 30 19:45:17.376978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.376959 4789 scope.go:122] "RemoveContainer" containerID="c41528be2ae1d56b23f27fc2a7bae21a57b90cfcadbc0971730da22230e14d0c" Oct 30 19:45:17.390817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.390798 4789 scope.go:122] "RemoveContainer" containerID="0687c47aa2386c57f90c5cd2d2cdb5fd7ce45164450f530023107d627034a6c9" Oct 30 19:45:17.407321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.407294 4789 scope.go:122] "RemoveContainer" containerID="250a708240efc8248c0d5539339eae10530dc0ba2721b68f7c41f426cdf75b8f" Oct 30 19:45:17.478817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.478758 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:17.610198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.610168 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:17.661855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.661832 4789 scope.go:122] "RemoveContainer" containerID="5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df" Oct 30 19:45:17.662483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.662015 4789 scope.go:122] "RemoveContainer" containerID="eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0" Oct 30 19:45:17.671487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.671463 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" path="/var/lib/kubelet/pods/6a18a7cf-efc4-451c-9d94-7d6213eaec43/volumes" Oct 30 19:45:17.704647 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.704618 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:17.947478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:17.947447 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:18.198245 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:18.198186 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:45:18.382444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.382418 4789 generic.go:356] "Generic (PLEG): container finished" podID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerID="9e251f6e64fcb0e946b68988fed960083f262be45b77c5aef057fa4788f81d3e" exitCode=60 Oct 30 19:45:18.382539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.382506 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerDied","Data":"9e251f6e64fcb0e946b68988fed960083f262be45b77c5aef057fa4788f81d3e"} Oct 30 19:45:18.382620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.382601 4789 scope.go:122] "RemoveContainer" containerID="c1b20116658c25c63559a2a6af4fcf9d0a2feabbf915011e99e9585cbfc8d4f4" Oct 30 19:45:18.383006 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.382984 4789 scope.go:122] "RemoveContainer" containerID="9e251f6e64fcb0e946b68988fed960083f262be45b77c5aef057fa4788f81d3e" Oct 30 19:45:18.383322 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:18.383296 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd_odh-ai-gateway-infra(f115a03b-628e-45a6-9e20-004f39e3b63a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" Oct 30 19:45:18.384465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.384446 4789 generic.go:356] "Generic (PLEG): container finished" podID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerID="2aebb40463993cbd194b5a032704c107fae325a6931a50314bb78b9c170fe7dd" exitCode=60 Oct 30 19:45:18.384565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.384518 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerDied","Data":"2aebb40463993cbd194b5a032704c107fae325a6931a50314bb78b9c170fe7dd"} Oct 30 19:45:18.384888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.384870 4789 scope.go:122] "RemoveContainer" containerID="2aebb40463993cbd194b5a032704c107fae325a6931a50314bb78b9c170fe7dd" Oct 30 19:45:18.385155 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:18.385133 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7_odh-ai-gateway-infra(1c7ca63d-d889-4bcc-bbcc-296d72c836b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" Oct 30 19:45:18.386315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.386298 4789 generic.go:356] "Generic (PLEG): container finished" podID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerID="e698c376248651bbda7d44c882cdf3736c6e751cba8222176410af8692748c7b" exitCode=60 Oct 30 19:45:18.386401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.386372 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerDied","Data":"e698c376248651bbda7d44c882cdf3736c6e751cba8222176410af8692748c7b"} Oct 30 19:45:18.386719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.386699 4789 scope.go:122] "RemoveContainer" containerID="e698c376248651bbda7d44c882cdf3736c6e751cba8222176410af8692748c7b" Oct 30 19:45:18.386958 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:18.386933 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-wpm77_odh-ai-gateway-infra(fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" Oct 30 19:45:18.388369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.388347 4789 generic.go:356] "Generic (PLEG): container finished" podID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerID="293e6f1e341290b96579bcffc2d664e4916d1eacc24e0a41ab9ee6624dbf5148" exitCode=60 Oct 30 19:45:18.388452 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.388417 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerDied","Data":"293e6f1e341290b96579bcffc2d664e4916d1eacc24e0a41ab9ee6624dbf5148"} Oct 30 19:45:18.388760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.388735 4789 scope.go:122] "RemoveContainer" containerID="293e6f1e341290b96579bcffc2d664e4916d1eacc24e0a41ab9ee6624dbf5148" Oct 30 19:45:18.388957 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:18.388937 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx_odh-ai-gateway-infra(c8a1e201-d495-43ed-bbf8-e2924186f420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" Oct 30 19:45:18.398915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.398891 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerStarted","Data":"921a43bbfb17a04c25922e779e4c08d7edd3de514c1f80153ae05114c1ab3108"} Oct 30 19:45:18.416754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.416717 4789 scope.go:122] "RemoveContainer" containerID="60819993123dc2cd12115b4f9b53734bc9d9f88227c57455662170412fc98333" Oct 30 19:45:18.457394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.454452 4789 scope.go:122] "RemoveContainer" containerID="a47a29d388672af44be2c2d11d3f009233296515d4a9e6d1bb44c1f6eb758603" Oct 30 19:45:18.476478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.476402 4789 scope.go:122] "RemoveContainer" containerID="de7c4e248c04ba3994a276fc7212e8d6a6eef1f0eff3a271bf609eacb9ac9d0a" Oct 30 19:45:18.663997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.663970 4789 scope.go:122] "RemoveContainer" containerID="6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463" Oct 30 19:45:18.666578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.666227 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:18.683399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.683372 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:18.703697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.702619 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:18.709799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.709779 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:18.765304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765279 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\") pod \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " Oct 30 19:45:18.765418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765342 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\") pod \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " Oct 30 19:45:18.765418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765393 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\") pod \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " Oct 30 19:45:18.765533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765483 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\") pod \"1fd465a2-3468-4755-af32-d0d22bbc6036\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " Oct 30 19:45:18.765631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765607 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\" (UniqueName: \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\") pod \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\" (UID: \"a6eb9bc0-34ac-47f6-97c8-c9734451f08c\") " Oct 30 19:45:18.765719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765661 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\" (UniqueName: \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\") pod \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\" (UID: \"ef8e6189-8748-4260-b2c2-51f2f07f11b4\") " Oct 30 19:45:18.765719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765690 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca" pod "a6eb9bc0-34ac-47f6-97c8-c9734451f08c" (UID: "a6eb9bc0-34ac-47f6-97c8-c9734451f08c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.765719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765696 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\") pod \"1fd465a2-3468-4755-af32-d0d22bbc6036\" (UID: \"1fd465a2-3468-4755-af32-d0d22bbc6036\") " Oct 30 19:45:18.765890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765788 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca" pod "1fd465a2-3468-4755-af32-d0d22bbc6036" (UID: "1fd465a2-3468-4755-af32-d0d22bbc6036"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.765890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765795 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca" pod "00ccb14e-6b93-41f5-bb36-df6ee7317e89" (UID: "00ccb14e-6b93-41f5-bb36-df6ee7317e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.765890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.765807 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\" (UniqueName: \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\") pod \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\" (UID: \"00ccb14e-6b93-41f5-bb36-df6ee7317e89\") " Oct 30 19:45:18.766062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.766011 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca" pod "ef8e6189-8748-4260-b2c2-51f2f07f11b4" (UID: "ef8e6189-8748-4260-b2c2-51f2f07f11b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:18.766454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.766436 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fd465a2-3468-4755-af32-d0d22bbc6036-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.766544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.766457 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8e6189-8748-4260-b2c2-51f2f07f11b4-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.766544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.766473 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.766544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.766487 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00ccb14e-6b93-41f5-bb36-df6ee7317e89-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.768464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.768443 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd" pod "a6eb9bc0-34ac-47f6-97c8-c9734451f08c" (UID: "a6eb9bc0-34ac-47f6-97c8-c9734451f08c"). InnerVolumeSpecName "kube-api-access-4v6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.768574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.768485 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m" pod "00ccb14e-6b93-41f5-bb36-df6ee7317e89" (UID: "00ccb14e-6b93-41f5-bb36-df6ee7317e89"). InnerVolumeSpecName "kube-api-access-rr29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.768574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.768513 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst" pod "ef8e6189-8748-4260-b2c2-51f2f07f11b4" (UID: "ef8e6189-8748-4260-b2c2-51f2f07f11b4"). InnerVolumeSpecName "kube-api-access-6rmst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.768871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.768857 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt" pod "1fd465a2-3468-4755-af32-d0d22bbc6036" (UID: "1fd465a2-3468-4755-af32-d0d22bbc6036"). InnerVolumeSpecName "kube-api-access-jr7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:18.867922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.867888 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6qd\" (UniqueName: \"kubernetes.io/projected/a6eb9bc0-34ac-47f6-97c8-c9734451f08c-kube-api-access-4v6qd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.868034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.867923 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rmst\" (UniqueName: \"kubernetes.io/projected/ef8e6189-8748-4260-b2c2-51f2f07f11b4-kube-api-access-6rmst\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.868034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.867939 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr7rt\" (UniqueName: \"kubernetes.io/projected/1fd465a2-3468-4755-af32-d0d22bbc6036-kube-api-access-jr7rt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:18.868034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:18.867953 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr29m\" (UniqueName: \"kubernetes.io/projected/00ccb14e-6b93-41f5-bb36-df6ee7317e89-kube-api-access-rr29m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.410555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.410448 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerStarted","Data":"3b5eb7eb77534a28d09d6a98feb569e8b0d50f86f843fdae9dc9abb4b00a9eba"} Oct 30 19:45:19.419811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.419776 4789 generic.go:356] "Generic (PLEG): container finished" podID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerID="78d167d0d4287223349c883cf3bcff2e93dd2239b5e87b9c6dedae3e57ad0537" exitCode=60 Oct 30 19:45:19.419961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.419807 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerDied","Data":"78d167d0d4287223349c883cf3bcff2e93dd2239b5e87b9c6dedae3e57ad0537"} Oct 30 19:45:19.419961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.419864 4789 scope.go:122] "RemoveContainer" containerID="5220d6d1cbf641ec92a75c69e1b843feeb4f833e7b8a84d79a844e1fc953a8df" Oct 30 19:45:19.420265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.420241 4789 scope.go:122] "RemoveContainer" containerID="78d167d0d4287223349c883cf3bcff2e93dd2239b5e87b9c6dedae3e57ad0537" Oct 30 19:45:19.420602 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:19.420576 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr_odh-ai-gateway-infra(6d893c93-da6c-44eb-8d06-81fd692e969e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" Oct 30 19:45:19.421280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.421258 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" event={"ID":"1fd465a2-3468-4755-af32-d0d22bbc6036","Type":"ContainerDied","Data":"e35c90aa57fb3a055075fc42b7d5879df5592cb1638af3bb51de8a75281b5661"} Oct 30 19:45:19.421384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.421267 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5" Oct 30 19:45:19.422682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.422662 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" event={"ID":"00ccb14e-6b93-41f5-bb36-df6ee7317e89","Type":"ContainerDied","Data":"d2c00939963856fc3d25e2283eaf4e797386ff29c713a138e36370c7784552cc"} Oct 30 19:45:19.422798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.422666 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9" Oct 30 19:45:19.424994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.424968 4789 generic.go:356] "Generic (PLEG): container finished" podID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerID="921a43bbfb17a04c25922e779e4c08d7edd3de514c1f80153ae05114c1ab3108" exitCode=60 Oct 30 19:45:19.425095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.425068 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerDied","Data":"921a43bbfb17a04c25922e779e4c08d7edd3de514c1f80153ae05114c1ab3108"} Oct 30 19:45:19.425466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.425444 4789 scope.go:122] "RemoveContainer" containerID="921a43bbfb17a04c25922e779e4c08d7edd3de514c1f80153ae05114c1ab3108" Oct 30 19:45:19.425765 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:19.425741 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp_odh-ai-gateway-infra(a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" Oct 30 19:45:19.426810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.426789 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" Oct 30 19:45:19.426810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.426803 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8" event={"ID":"ef8e6189-8748-4260-b2c2-51f2f07f11b4","Type":"ContainerDied","Data":"8972852ff08c6ed691387235f3acab58a41c8f38787476a2fe2b36c9f1441b81"} Oct 30 19:45:19.429346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.429322 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" event={"ID":"a6eb9bc0-34ac-47f6-97c8-c9734451f08c","Type":"ContainerDied","Data":"0e4af09f2509306157ff24d2db0099fe48c64dc9a1c9440457a12eb9179d27e3"} Oct 30 19:45:19.429450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.429426 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s" Oct 30 19:45:19.437969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.437951 4789 scope.go:122] "RemoveContainer" containerID="6f362e4648b5f7281b92eb660a27c5d1a13a46fe1a547a62753f0d5d3ea325d5" Oct 30 19:45:19.449110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.449090 4789 scope.go:122] "RemoveContainer" containerID="e872320d9ba7998e36325f4dc6efb94bfd071864d6e10aa7e543d09a74491a3e" Oct 30 19:45:19.458554 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.458538 4789 scope.go:122] "RemoveContainer" containerID="eb7d7587f2b10e7f6f13a36643eac274127db16b8f28014ff276ce26d15133c0" Oct 30 19:45:19.470969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.470940 4789 scope.go:122] "RemoveContainer" containerID="077cb8f086521370869c8d08d7125791232668a0c9702a72889a364927cd2a3f" Oct 30 19:45:19.479489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.479469 4789 scope.go:122] "RemoveContainer" containerID="3b0c3242d4277967394fed02f7257199a1acf4b6d0dfeb8b4226ee8959c6f2cb" Oct 30 19:45:19.755016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.754988 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:19.858952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:19.858921 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:20.032513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.032482 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:20.082106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.082078 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\") pod \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " Oct 30 19:45:20.082309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.082295 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\" (UniqueName: \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\") pod \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\" (UID: \"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b\") " Oct 30 19:45:20.082445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.082422 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca" pod "fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" (UID: "fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.082575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.082561 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.084371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.084344 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg" pod "fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" (UID: "fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b"). InnerVolumeSpecName "kube-api-access-8d4gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.089979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.089949 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:20.115710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.115671 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:20.143628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.143563 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-85f471-29889825-7c8w8"] Oct 30 19:45:20.184225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.184181 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4gg\" (UniqueName: \"kubernetes.io/projected/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b-kube-api-access-8d4gg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.247898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.247828 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:20.252821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.252793 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-db06cf-29889825-mhq5s"] Oct 30 19:45:20.264798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.264777 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:20.315763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.315730 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:20.360719 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.360688 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:20.365436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.365400 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b2604c-29889825-4wms9"] Oct 30 19:45:20.389784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.389741 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\" (UniqueName: \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\") pod \"f115a03b-628e-45a6-9e20-004f39e3b63a\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " Oct 30 19:45:20.389926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.389886 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\") pod \"f115a03b-628e-45a6-9e20-004f39e3b63a\" (UID: \"f115a03b-628e-45a6-9e20-004f39e3b63a\") " Oct 30 19:45:20.390540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.390511 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca" pod "f115a03b-628e-45a6-9e20-004f39e3b63a" (UID: "f115a03b-628e-45a6-9e20-004f39e3b63a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.393153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.393077 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg" pod "f115a03b-628e-45a6-9e20-004f39e3b63a" (UID: "f115a03b-628e-45a6-9e20-004f39e3b63a"). InnerVolumeSpecName "kube-api-access-svshg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.434117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.434086 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:20.448896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.448860 4789 generic.go:356] "Generic (PLEG): container finished" podID="934493f5-bc4a-4863-8273-824889175a5a" containerID="3b5eb7eb77534a28d09d6a98feb569e8b0d50f86f843fdae9dc9abb4b00a9eba" exitCode=60 Oct 30 19:45:20.449068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.448936 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerDied","Data":"3b5eb7eb77534a28d09d6a98feb569e8b0d50f86f843fdae9dc9abb4b00a9eba"} Oct 30 19:45:20.449068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.449020 4789 scope.go:122] "RemoveContainer" containerID="6508198addc8280930fbc8dcd5936c85e6407802c8db6f871fe61f93f34c6463" Oct 30 19:45:20.449466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.449436 4789 scope.go:122] "RemoveContainer" containerID="3b5eb7eb77534a28d09d6a98feb569e8b0d50f86f843fdae9dc9abb4b00a9eba" Oct 30 19:45:20.449834 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:20.449810 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh_odh-ai-gateway-infra(934493f5-bc4a-4863-8273-824889175a5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" podUID="934493f5-bc4a-4863-8273-824889175a5a" Oct 30 19:45:20.451181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.451161 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" Oct 30 19:45:20.451306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.451165 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd" event={"ID":"f115a03b-628e-45a6-9e20-004f39e3b63a","Type":"ContainerDied","Data":"c826287be475cdc725780500deaf49d9e6687226be46394f7ce8eb990e006d54"} Oct 30 19:45:20.453948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.453928 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" Oct 30 19:45:20.454039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.453984 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77" event={"ID":"fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b","Type":"ContainerDied","Data":"ef7045c774054dd15a974ccc1eb1b18625e1eff52af0c1454f099312a22dd498"} Oct 30 19:45:20.461770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.461747 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:20.468394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.468365 4789 scope.go:122] "RemoveContainer" containerID="9e251f6e64fcb0e946b68988fed960083f262be45b77c5aef057fa4788f81d3e" Oct 30 19:45:20.482526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.482490 4789 scope.go:122] "RemoveContainer" containerID="e698c376248651bbda7d44c882cdf3736c6e751cba8222176410af8692748c7b" Oct 30 19:45:20.492047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.492007 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svshg\" (UniqueName: \"kubernetes.io/projected/f115a03b-628e-45a6-9e20-004f39e3b63a-kube-api-access-svshg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.492047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.492048 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f115a03b-628e-45a6-9e20-004f39e3b63a-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.593250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.593189 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\") pod \"c8a1e201-d495-43ed-bbf8-e2924186f420\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " Oct 30 19:45:20.593425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.593257 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\" (UniqueName: \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\") pod \"c8a1e201-d495-43ed-bbf8-e2924186f420\" (UID: \"c8a1e201-d495-43ed-bbf8-e2924186f420\") " Oct 30 19:45:20.594329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.594306 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca" pod "c8a1e201-d495-43ed-bbf8-e2924186f420" (UID: "c8a1e201-d495-43ed-bbf8-e2924186f420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.606638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.602251 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f" pod "c8a1e201-d495-43ed-bbf8-e2924186f420" (UID: "c8a1e201-d495-43ed-bbf8-e2924186f420"). InnerVolumeSpecName "kube-api-access-6q69f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.612959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.612931 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:20.659316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.659289 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:20.694177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694148 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\") pod \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " Oct 30 19:45:20.694390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694372 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\" (UniqueName: \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\") pod \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\" (UID: \"1c7ca63d-d889-4bcc-bbcc-296d72c836b2\") " Oct 30 19:45:20.694558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694531 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca" pod "1c7ca63d-d889-4bcc-bbcc-296d72c836b2" (UID: "1c7ca63d-d889-4bcc-bbcc-296d72c836b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.694955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694937 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.695014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694961 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a1e201-d495-43ed-bbf8-e2924186f420-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.695014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.694978 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q69f\" (UniqueName: \"kubernetes.io/projected/c8a1e201-d495-43ed-bbf8-e2924186f420-kube-api-access-6q69f\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.696416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.696395 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb" pod "1c7ca63d-d889-4bcc-bbcc-296d72c836b2" (UID: "1c7ca63d-d889-4bcc-bbcc-296d72c836b2"). InnerVolumeSpecName "kube-api-access-tfrwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.796076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.796005 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\" (UniqueName: \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\") pod \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " Oct 30 19:45:20.796466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.796173 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\") pod \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\" (UID: \"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5\") " Oct 30 19:45:20.796526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.796504 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca" pod "a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" (UID: "a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:20.796684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.796668 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.796753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.796687 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfrwb\" (UniqueName: \"kubernetes.io/projected/1c7ca63d-d889-4bcc-bbcc-296d72c836b2-kube-api-access-tfrwb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:20.798068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.798048 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt" pod "a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" (UID: "a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5"). InnerVolumeSpecName "kube-api-access-mrndt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:20.807582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.807559 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:20.813557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.813527 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-af72cf-29889825-4kcs5"] Oct 30 19:45:20.902318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.902070 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:20.903542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:20.903516 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrndt\" (UniqueName: \"kubernetes.io/projected/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5-kube-api-access-mrndt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.422078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.422040 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:21.428635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.428607 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-wpm77"] Oct 30 19:45:21.474437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.474410 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" Oct 30 19:45:21.474578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.474434 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx" event={"ID":"c8a1e201-d495-43ed-bbf8-e2924186f420","Type":"ContainerDied","Data":"77e3812cdd9ce84fc681194c1be2298d2304507c212594bef4cfa129791d41ce"} Oct 30 19:45:21.474578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.474489 4789 scope.go:122] "RemoveContainer" containerID="293e6f1e341290b96579bcffc2d664e4916d1eacc24e0a41ab9ee6624dbf5148" Oct 30 19:45:21.475853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.475838 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" Oct 30 19:45:21.475948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.475871 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp" event={"ID":"a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5","Type":"ContainerDied","Data":"e1237574d6c12da96cb412bed5eb228a64fe0171e24dd495b12e68e07a803255"} Oct 30 19:45:21.480557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.480533 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" event={"ID":"1c7ca63d-d889-4bcc-bbcc-296d72c836b2","Type":"ContainerDied","Data":"e1bc8e5b98ff94e0cfeeb04c9157d516f97c1f73df2deaa95face8170afd8b15"} Oct 30 19:45:21.480661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.480570 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7" Oct 30 19:45:21.491015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.490996 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:21.492052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.491798 4789 scope.go:122] "RemoveContainer" containerID="921a43bbfb17a04c25922e779e4c08d7edd3de514c1f80153ae05114c1ab3108" Oct 30 19:45:21.506489 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.506450 4789 scope.go:122] "RemoveContainer" containerID="2aebb40463993cbd194b5a032704c107fae325a6931a50314bb78b9c170fe7dd" Oct 30 19:45:21.610932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.610355 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:21.613535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.613510 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-a2f378-29889825-2q5bd"] Oct 30 19:45:21.684136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.683966 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" path="/var/lib/kubelet/pods/00ccb14e-6b93-41f5-bb36-df6ee7317e89/volumes" Oct 30 19:45:21.684712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.684690 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" path="/var/lib/kubelet/pods/1fd465a2-3468-4755-af32-d0d22bbc6036/volumes" Oct 30 19:45:21.685318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.685300 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" path="/var/lib/kubelet/pods/a6eb9bc0-34ac-47f6-97c8-c9734451f08c/volumes" Oct 30 19:45:21.686265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.686246 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" path="/var/lib/kubelet/pods/ef8e6189-8748-4260-b2c2-51f2f07f11b4/volumes" Oct 30 19:45:21.686787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.686770 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" path="/var/lib/kubelet/pods/f115a03b-628e-45a6-9e20-004f39e3b63a/volumes" Oct 30 19:45:21.687317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.687300 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" path="/var/lib/kubelet/pods/fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b/volumes" Oct 30 19:45:21.687629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.687612 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:21.705725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.705702 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:21.815321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.815290 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\" (UniqueName: \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\") pod \"934493f5-bc4a-4863-8273-824889175a5a\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " Oct 30 19:45:21.815787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.815348 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\" (UniqueName: \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\") pod \"6d893c93-da6c-44eb-8d06-81fd692e969e\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " Oct 30 19:45:21.815787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.815392 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\") pod \"6d893c93-da6c-44eb-8d06-81fd692e969e\" (UID: \"6d893c93-da6c-44eb-8d06-81fd692e969e\") " Oct 30 19:45:21.815787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.815451 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\") pod \"934493f5-bc4a-4863-8273-824889175a5a\" (UID: \"934493f5-bc4a-4863-8273-824889175a5a\") " Oct 30 19:45:21.816047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.816020 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca" pod "6d893c93-da6c-44eb-8d06-81fd692e969e" (UID: "6d893c93-da6c-44eb-8d06-81fd692e969e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.816155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.816137 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d893c93-da6c-44eb-8d06-81fd692e969e-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.816720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.816691 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca" pod "934493f5-bc4a-4863-8273-824889175a5a" (UID: "934493f5-bc4a-4863-8273-824889175a5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.819063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.819025 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl" pod "6d893c93-da6c-44eb-8d06-81fd692e969e" (UID: "6d893c93-da6c-44eb-8d06-81fd692e969e"). InnerVolumeSpecName "kube-api-access-4pqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.819430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.819399 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc" pod "934493f5-bc4a-4863-8273-824889175a5a" (UID: "934493f5-bc4a-4863-8273-824889175a5a"). InnerVolumeSpecName "kube-api-access-tfnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.831046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.831020 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:21.834802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.834783 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-22fb18-29889825-md8bp"] Oct 30 19:45:21.917895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.917864 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnpc\" (UniqueName: \"kubernetes.io/projected/934493f5-bc4a-4863-8273-824889175a5a-kube-api-access-tfnpc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.917895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.917891 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pqxl\" (UniqueName: \"kubernetes.io/projected/6d893c93-da6c-44eb-8d06-81fd692e969e-kube-api-access-4pqxl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.918140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.917904 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934493f5-bc4a-4863-8273-824889175a5a-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.941812 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.941741 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:21.944959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:21.944934 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-9275c3-29889825-mjnsx"] Oct 30 19:45:22.049462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.049427 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:22.052273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.052237 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-2301cb-29889825-vmcj7"] Oct 30 19:45:22.489453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.489426 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" Oct 30 19:45:22.489628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.489450 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh" event={"ID":"934493f5-bc4a-4863-8273-824889175a5a","Type":"ContainerDied","Data":"981edec9a2155bacc44fc9ec5ae386ba153075ea9b8d04a6fecee321789b0a6f"} Oct 30 19:45:22.489628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.489519 4789 scope.go:122] "RemoveContainer" containerID="3b5eb7eb77534a28d09d6a98feb569e8b0d50f86f843fdae9dc9abb4b00a9eba" Oct 30 19:45:22.493191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.493166 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" event={"ID":"6d893c93-da6c-44eb-8d06-81fd692e969e","Type":"ContainerDied","Data":"720fe2f22cb6d58da78f51b3a83aabf98ffb778d15f169176f47d794407a9a8f"} Oct 30 19:45:22.493328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.493202 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr" Oct 30 19:45:22.501351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.501330 4789 scope.go:122] "RemoveContainer" containerID="78d167d0d4287223349c883cf3bcff2e93dd2239b5e87b9c6dedae3e57ad0537" Oct 30 19:45:22.644469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.644431 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:22.648449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.648424 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d6f58f-29889825-4grhh"] Oct 30 19:45:22.752966 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.752893 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:22.762436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:22.762403 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2186d8-29889825-rghqr"] Oct 30 19:45:23.670657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:23.670619 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" path="/var/lib/kubelet/pods/1c7ca63d-d889-4bcc-bbcc-296d72c836b2/volumes" Oct 30 19:45:23.671044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:23.671028 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" path="/var/lib/kubelet/pods/6d893c93-da6c-44eb-8d06-81fd692e969e/volumes" Oct 30 19:45:23.671420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:23.671398 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934493f5-bc4a-4863-8273-824889175a5a" path="/var/lib/kubelet/pods/934493f5-bc4a-4863-8273-824889175a5a/volumes" Oct 30 19:45:23.672076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:23.672058 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" path="/var/lib/kubelet/pods/a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5/volumes" Oct 30 19:45:23.672446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:23.672431 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" path="/var/lib/kubelet/pods/c8a1e201-d495-43ed-bbf8-e2924186f420/volumes" Oct 30 19:45:31.018583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:31.018540 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:45:31.019496 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:31.018785 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" containerID="cri-o://606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62" gracePeriod=30 Oct 30 19:45:35.526565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:35.526528 4789 patch_prober.go:29] interesting pod/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:45:35.527038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:35.526595 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:45:36.289336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.289271 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:45:36.378880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.378844 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.378896 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.378919 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.378937 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.378974 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379066 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379129 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379143 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:36.379345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379185 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379265 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\") pod \"d1812787-bdf2-4b94-84d0-095bfe4ba703\" (UID: \"d1812787-bdf2-4b94-84d0-095bfe4ba703\") " Oct 30 19:45:36.379635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379399 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:36.379635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379471 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:36.379635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379574 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:36.379801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379695 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:36.379839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379819 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.379872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379842 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.379872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379857 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1812787-bdf2-4b94-84d0-095bfe4ba703-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.379937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379872 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.379937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.379887 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.381249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.381224 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:36.381382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.381304 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "kube-api-access-nzbz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:36.381600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.381582 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:45:36.381670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.381602 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token" pod "d1812787-bdf2-4b94-84d0-095bfe4ba703" (UID: "d1812787-bdf2-4b94-84d0-095bfe4ba703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:36.480588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.480548 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.480588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.480576 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbz9\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-kube-api-access-nzbz9\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.480588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.480586 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.480909 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.480597 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1812787-bdf2-4b94-84d0-095bfe4ba703-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.585420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.585334 4789 generic.go:356] "Generic (PLEG): container finished" podID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerID="606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62" exitCode=0 Oct 30 19:45:36.585420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.585393 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" Oct 30 19:45:36.585896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.585420 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" event={"ID":"d1812787-bdf2-4b94-84d0-095bfe4ba703","Type":"ContainerDied","Data":"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62"} Oct 30 19:45:36.585896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.585477 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz" event={"ID":"d1812787-bdf2-4b94-84d0-095bfe4ba703","Type":"ContainerDied","Data":"c28305814f7bbf67cad1c10aa6befe0084213f64f2e89f9076acbb3ff4f9112c"} Oct 30 19:45:36.585896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.585509 4789 scope.go:122] "RemoveContainer" containerID="606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62" Oct 30 19:45:36.596674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.596651 4789 scope.go:122] "RemoveContainer" containerID="606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62" Oct 30 19:45:36.596996 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:36.596971 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62\": container with ID starting with 606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62 not found: ID does not exist" containerID="606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62" Oct 30 19:45:36.597087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.597005 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62"} err="failed to get container status \"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62\": rpc error: code = NotFound desc = could not find container \"606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62\": container with ID starting with 606163c8331e4b1ad0a9b3f48189d1879f9778da853050e39802cf033b56dd62 not found: ID does not exist" Oct 30 19:45:36.743934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.743900 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:45:36.749328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:36.749305 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9275c3-openshift-default-56ffdfd665-2xtdz"] Oct 30 19:45:37.669426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:45:37.669393 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" path="/var/lib/kubelet/pods/d1812787-bdf2-4b94-84d0-095bfe4ba703/volumes" Oct 30 19:45:48.196721 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:45:48.196682 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:46:04.524436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:04.524406 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:46:04.538635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:04.538611 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:46:18.196936 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:46:18.196897 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:46:29.245790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.245753 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw"] Oct 30 19:46:29.246309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246295 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246310 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246322 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246328 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246336 4789 cpu_manager.go:414] "containerMap: removing container" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246342 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246348 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246354 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246362 4789 cpu_manager.go:414] "containerMap: removing container" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246367 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246374 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.246375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246379 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246387 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246391 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246399 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246404 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246413 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246417 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246424 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246429 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246435 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246439 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246448 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246452 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246462 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246466 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246473 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246477 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246485 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246489 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246499 4789 cpu_manager.go:414] "containerMap: removing container" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246503 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246511 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246517 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246524 4789 cpu_manager.go:414] "containerMap: removing container" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246528 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246535 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246538 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246544 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246548 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246558 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246562 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246572 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246577 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246588 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246593 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246601 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246606 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246615 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246620 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246626 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246630 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246711 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a18a7cf-efc4-451c-9d94-7d6213eaec43" containerName="istio-proxy" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246719 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246726 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246731 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246738 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246745 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246750 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246756 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7ca63d-d889-4bcc-bbcc-296d72c836b2" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246761 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246767 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.246753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246773 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246779 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246783 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246788 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246792 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246797 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246803 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246808 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246814 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246819 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246825 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246831 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246837 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246840 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246847 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a1e201-d495-43ed-bbf8-e2924186f420" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246853 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246859 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246865 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1812787-bdf2-4b94-84d0-095bfe4ba703" containerName="istio-proxy" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246870 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:46:29.248337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.246875 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:46:29.251902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.251883 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.306385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.306359 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:46:29.306994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.306971 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:46:29.352904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.352870 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:46:29.355898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.355869 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw"] Oct 30 19:46:29.356047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.355869 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.356047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.355970 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46dt\" (UniqueName: \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.356047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.356014 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.405362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.405340 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:46:29.456791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.456764 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.456899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.456803 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46dt\" (UniqueName: \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.456899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.456837 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.457439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.457418 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.459168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.459147 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.515596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.515537 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46dt\" (UniqueName: \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\") pod \"maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.564479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.564452 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:29.948466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.947046 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw"] Oct 30 19:46:29.969884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:29.969856 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" event={"ID":"ad732af7-2b5b-4354-bb73-29d403a34d83","Type":"ContainerStarted","Data":"d07ae7036b7c3cd611c0eb2ecdb3aaf827c60017609192ea6a3b0cc4b7f8a545"} Oct 30 19:46:30.977835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:30.977749 4789 generic.go:356] "Generic (PLEG): container finished" podID="ad732af7-2b5b-4354-bb73-29d403a34d83" containerID="2b5f56d4085df1ccd964a5cb8241d869f73b530b0b5d1c620265298caa5f50b2" exitCode=0 Oct 30 19:46:30.978247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:30.977827 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" event={"ID":"ad732af7-2b5b-4354-bb73-29d403a34d83","Type":"ContainerDied","Data":"2b5f56d4085df1ccd964a5cb8241d869f73b530b0b5d1c620265298caa5f50b2"} Oct 30 19:46:32.141188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.141155 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:32.286381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.286317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\" (UniqueName: \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\") pod \"ad732af7-2b5b-4354-bb73-29d403a34d83\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " Oct 30 19:46:32.286381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.286364 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\") pod \"ad732af7-2b5b-4354-bb73-29d403a34d83\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " Oct 30 19:46:32.286572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.286432 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\") pod \"ad732af7-2b5b-4354-bb73-29d403a34d83\" (UID: \"ad732af7-2b5b-4354-bb73-29d403a34d83\") " Oct 30 19:46:32.286878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.286838 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca" pod "ad732af7-2b5b-4354-bb73-29d403a34d83" (UID: "ad732af7-2b5b-4354-bb73-29d403a34d83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:32.287325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.287256 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:32.288617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.288596 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca" pod "ad732af7-2b5b-4354-bb73-29d403a34d83" (UID: "ad732af7-2b5b-4354-bb73-29d403a34d83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:32.289032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.289017 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt" pod "ad732af7-2b5b-4354-bb73-29d403a34d83" (UID: "ad732af7-2b5b-4354-bb73-29d403a34d83"). InnerVolumeSpecName "kube-api-access-t46dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:32.388027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.387999 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad732af7-2b5b-4354-bb73-29d403a34d83-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:32.388027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.388027 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46dt\" (UniqueName: \"kubernetes.io/projected/ad732af7-2b5b-4354-bb73-29d403a34d83-kube-api-access-t46dt\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:32.991498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.991460 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" Oct 30 19:46:32.991498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.991492 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw" event={"ID":"ad732af7-2b5b-4354-bb73-29d403a34d83","Type":"ContainerDied","Data":"d07ae7036b7c3cd611c0eb2ecdb3aaf827c60017609192ea6a3b0cc4b7f8a545"} Oct 30 19:46:32.991767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:32.991539 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07ae7036b7c3cd611c0eb2ecdb3aaf827c60017609192ea6a3b0cc4b7f8a545" Oct 30 19:46:40.741170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:40.741128 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:46:45.742138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:45.742058 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" containerID="cri-o://9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225" gracePeriod=25 Oct 30 19:46:46.771734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:46.771692 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:46:48.196613 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:46:48.196580 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:46:56.771486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:46:56.771447 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:47:06.019411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.019383 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:47:06.117938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.117913 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5"] Oct 30 19:47:06.118506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118490 4789 cpu_manager.go:414] "containerMap: removing container" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118509 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118521 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118526 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eb9bc0-34ac-47f6-97c8-c9734451f08c" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118539 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118543 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8e6189-8748-4260-b2c2-51f2f07f11b4" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118554 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:47:06.118558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118558 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5c8872-8f4b-4d05-a1f1-7a29c7789a9b" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118578 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118587 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118609 4789 cpu_manager.go:414] "containerMap: removing container" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118614 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118623 4789 cpu_manager.go:414] "containerMap: removing container" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118628 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118635 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118639 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd465a2-3468-4755-af32-d0d22bbc6036" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118645 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118649 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118664 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118671 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118689 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ad732af7-2b5b-4354-bb73-29d403a34d83" containerName="revoke-keys" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118693 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad732af7-2b5b-4354-bb73-29d403a34d83" containerName="revoke-keys" Oct 30 19:47:06.118778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118779 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9da7fe9-553d-43ab-8c1e-d8bc607dc4b5" containerName="cleanup" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118789 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad732af7-2b5b-4354-bb73-29d403a34d83" containerName="revoke-keys" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118796 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d893c93-da6c-44eb-8d06-81fd692e969e" containerName="cleanup" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118804 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f115a03b-628e-45a6-9e20-004f39e3b63a" containerName="cleanup" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118815 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerName="payload-pre-processing" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118822 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ccb14e-6b93-41f5-bb36-df6ee7317e89" containerName="cleanup" Oct 30 19:47:06.119283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.118828 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="934493f5-bc4a-4863-8273-824889175a5a" containerName="cleanup" Oct 30 19:47:06.122193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.122177 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.157572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.157545 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5"] Oct 30 19:47:06.169244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.169182 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\" (UniqueName: \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\") pod \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " Oct 30 19:47:06.169376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.169285 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\") pod \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\" (UID: \"d7c017e7-55aa-49ed-bb53-3f12516a8d67\") " Oct 30 19:47:06.169742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.169711 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume" pod "d7c017e7-55aa-49ed-bb53-3f12516a8d67" (UID: "d7c017e7-55aa-49ed-bb53-3f12516a8d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:06.172063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.172036 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4" pod "d7c017e7-55aa-49ed-bb53-3f12516a8d67" (UID: "d7c017e7-55aa-49ed-bb53-3f12516a8d67"). InnerVolumeSpecName "kube-api-access-n72n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:06.172292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.172269 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:47:06.180152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.180132 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:06.180442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.180134 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:47:06.180556 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.180180 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:47:06.232685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.232663 4789 generic.go:356] "Generic (PLEG): container finished" podID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" containerID="9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225" exitCode=0 Oct 30 19:47:06.232806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.232705 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" Oct 30 19:47:06.232806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.232728 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" event={"ID":"d7c017e7-55aa-49ed-bb53-3f12516a8d67","Type":"ContainerDied","Data":"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225"} Oct 30 19:47:06.232806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.232786 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw" event={"ID":"d7c017e7-55aa-49ed-bb53-3f12516a8d67","Type":"ContainerDied","Data":"b948ff394b1c4549b8630a2227ab061b1731b8ae33e06a8b9f24c68859ad631d"} Oct 30 19:47:06.232937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.232812 4789 scope.go:122] "RemoveContainer" containerID="9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225" Oct 30 19:47:06.243740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.243642 4789 scope.go:122] "RemoveContainer" containerID="9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225" Oct 30 19:47:06.243991 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:06.243925 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225\": container with ID starting with 9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225 not found: ID does not exist" containerID="9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225" Oct 30 19:47:06.243991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.243963 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225"} err="failed to get container status \"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225\": rpc error: code = NotFound desc = could not find container \"9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225\": container with ID starting with 9b0458f665c74ba6fb6152f9b061e6084e390dafe16e731bdd84888b508eb225 not found: ID does not exist" Oct 30 19:47:06.270630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.270582 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.270720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.270681 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.270762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.270739 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsrq\" (UniqueName: \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.270820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.270801 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72n4\" (UniqueName: \"kubernetes.io/projected/d7c017e7-55aa-49ed-bb53-3f12516a8d67-kube-api-access-n72n4\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:06.270916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.270822 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7c017e7-55aa-49ed-bb53-3f12516a8d67-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:06.371528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.371501 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.371707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.371565 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsrq\" (UniqueName: \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.371783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.371711 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.372336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.372308 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.373727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.373706 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.401243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.401203 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:47:06.407163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.407135 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-af72cf-757746979-mthgw"] Oct 30 19:47:06.429663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.429632 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsrq\" (UniqueName: \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\") pod \"maas-api-revoke-sub-33764595-w2vk5\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.435284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.435265 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:06.808193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:06.808164 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5"] Oct 30 19:47:06.810263 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:06.810237 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12233c2a_55ed_49d6_a9fd_0a676592b459.slice/crio-f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32 WatchSource:0}: Error finding container f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32: Status 404 returned error can't find the container with id f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32 Oct 30 19:47:07.239620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:07.239588 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" event={"ID":"12233c2a-55ed-49d6-a9fd-0a676592b459","Type":"ContainerStarted","Data":"710074b3ccf51260c1adc957a35f93961f81a9b0064ea102a2b34221084f809a"} Oct 30 19:47:07.240012 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:07.239642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" event={"ID":"12233c2a-55ed-49d6-a9fd-0a676592b459","Type":"ContainerStarted","Data":"f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32"} Oct 30 19:47:07.351901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:07.351820 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" podStartSLOduration=1.351807543 podStartE2EDuration="1.351807543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:07.349387044 +0000 UTC m=+3663.851297529" watchObservedRunningTime="2026-10-30 19:47:07.351807543 +0000 UTC m=+3663.853718028" Oct 30 19:47:07.669781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:07.669756 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c017e7-55aa-49ed-bb53-3f12516a8d67" path="/var/lib/kubelet/pods/d7c017e7-55aa-49ed-bb53-3f12516a8d67/volumes" Oct 30 19:47:08.246866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:08.246825 4789 generic.go:356] "Generic (PLEG): container finished" podID="12233c2a-55ed-49d6-a9fd-0a676592b459" containerID="710074b3ccf51260c1adc957a35f93961f81a9b0064ea102a2b34221084f809a" exitCode=0 Oct 30 19:47:08.247314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:08.246881 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" event={"ID":"12233c2a-55ed-49d6-a9fd-0a676592b459","Type":"ContainerDied","Data":"710074b3ccf51260c1adc957a35f93961f81a9b0064ea102a2b34221084f809a"} Oct 30 19:47:09.414406 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.414376 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:09.605911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.605831 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\") pod \"12233c2a-55ed-49d6-a9fd-0a676592b459\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " Oct 30 19:47:09.606077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.605926 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\") pod \"12233c2a-55ed-49d6-a9fd-0a676592b459\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " Oct 30 19:47:09.606160 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.606106 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\" (UniqueName: \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\") pod \"12233c2a-55ed-49d6-a9fd-0a676592b459\" (UID: \"12233c2a-55ed-49d6-a9fd-0a676592b459\") " Oct 30 19:47:09.606261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.606205 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca" pod "12233c2a-55ed-49d6-a9fd-0a676592b459" (UID: "12233c2a-55ed-49d6-a9fd-0a676592b459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:09.608350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.608320 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:09.609058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.609002 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca" pod "12233c2a-55ed-49d6-a9fd-0a676592b459" (UID: "12233c2a-55ed-49d6-a9fd-0a676592b459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:09.620823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.620794 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq" pod "12233c2a-55ed-49d6-a9fd-0a676592b459" (UID: "12233c2a-55ed-49d6-a9fd-0a676592b459"). InnerVolumeSpecName "kube-api-access-9xsrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:09.709522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.709497 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsrq\" (UniqueName: \"kubernetes.io/projected/12233c2a-55ed-49d6-a9fd-0a676592b459-kube-api-access-9xsrq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:09.709522 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:09.709521 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12233c2a-55ed-49d6-a9fd-0a676592b459-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:10.260929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:10.260899 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" Oct 30 19:47:10.260929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:10.260905 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5" event={"ID":"12233c2a-55ed-49d6-a9fd-0a676592b459","Type":"ContainerDied","Data":"f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32"} Oct 30 19:47:10.261195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:10.260968 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f10a52fe819e5b164e8b6808d02b41700809d5e12207c03ae6e2bd16ac1e6f32" Oct 30 19:47:11.672065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:11.672034 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:47:11.672712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:11.672267 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" containerID="cri-o://df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7" gracePeriod=30 Oct 30 19:47:15.453239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:15.453185 4789 patch_prober.go:29] interesting pod/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:47:15.453694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:15.453267 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:16.810840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.810809 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb"] Oct 30 19:47:16.811445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.811416 4789 cpu_manager.go:414] "containerMap: removing container" podUID="12233c2a-55ed-49d6-a9fd-0a676592b459" containerName="revoke-keys" Oct 30 19:47:16.811445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.811433 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12233c2a-55ed-49d6-a9fd-0a676592b459" containerName="revoke-keys" Oct 30 19:47:16.811590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.811566 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="12233c2a-55ed-49d6-a9fd-0a676592b459" containerName="revoke-keys" Oct 30 19:47:16.814858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.814838 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.884164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.884113 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.884413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.884173 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8lq\" (UniqueName: \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.884413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.884367 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.912757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.911972 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:16.912757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.912037 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:47:16.912757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.912104 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:47:16.912757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.912149 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:47:16.917159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.917125 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb"] Oct 30 19:47:16.962619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.962600 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:47:16.985622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985596 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985660 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985677 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985705 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985737 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985798 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.985879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985840 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:16.986042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985863 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.986042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985912 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.986042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.985951 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:16.986232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986054 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:16.986232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986162 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:16.986340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986318 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\") pod \"b0370e1f-45da-4c11-b695-80c2db8aca48\" (UID: \"b0370e1f-45da-4c11-b695-80c2db8aca48\") " Oct 30 19:47:16.986616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986590 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:16.986713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986679 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.986791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986755 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8lq\" (UniqueName: \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.986868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.986849 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.987470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987430 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:16.987584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987564 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:16.987659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987589 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:16.987659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987607 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0370e1f-45da-4c11-b695-80c2db8aca48-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:16.987943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987918 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:16.988051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.987950 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:16.988537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.988504 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:16.988629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.988544 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "kube-api-access-rxxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:16.988629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.988618 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:16.988716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.988687 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token" pod "b0370e1f-45da-4c11-b695-80c2db8aca48" (UID: "b0370e1f-45da-4c11-b695-80c2db8aca48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:16.989448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:16.989431 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:17.044500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.044473 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8lq\" (UniqueName: \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\") pod \"maas-api-revoke-sub-bf3c95be-tbprb\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:17.088930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.088911 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxpd\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-kube-api-access-rxxpd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.088930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.088932 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.089059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.088941 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.089059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.088950 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0370e1f-45da-4c11-b695-80c2db8aca48-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.131090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.131067 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:17.306112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.306072 4789 generic.go:356] "Generic (PLEG): container finished" podID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerID="df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7" exitCode=0 Oct 30 19:47:17.306312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.306130 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" Oct 30 19:47:17.306312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.306160 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" event={"ID":"b0370e1f-45da-4c11-b695-80c2db8aca48","Type":"ContainerDied","Data":"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7"} Oct 30 19:47:17.306312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.306233 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd" event={"ID":"b0370e1f-45da-4c11-b695-80c2db8aca48","Type":"ContainerDied","Data":"1d72dfbb01f493f3a73d020272cac92c0bc1fa4e166605ecbfd477aedb549e14"} Oct 30 19:47:17.306312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.306262 4789 scope.go:122] "RemoveContainer" containerID="df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7" Oct 30 19:47:17.317578 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.317457 4789 scope.go:122] "RemoveContainer" containerID="df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7" Oct 30 19:47:17.317986 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:17.317903 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7\": container with ID starting with df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7 not found: ID does not exist" containerID="df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7" Oct 30 19:47:17.317986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.317938 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7"} err="failed to get container status \"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7\": rpc error: code = NotFound desc = could not find container \"df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7\": container with ID starting with df98015853805e51d573e1fa9f78bc6ad515c23e28c366b913c4f13c120a76b7 not found: ID does not exist" Oct 30 19:47:17.432105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.432063 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:47:17.432355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.432321 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" containerID="cri-o://2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e" gracePeriod=30 Oct 30 19:47:17.477097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.471978 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:47:17.481418 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.481394 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-330025-openshift-default-5c47dd8bc5-nbddd"] Oct 30 19:47:17.540283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.540256 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb"] Oct 30 19:47:17.543142 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:17.543114 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda61215e_da45_4fd6_88b9_892e00d91696.slice/crio-c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb WatchSource:0}: Error finding container c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb: Status 404 returned error can't find the container with id c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb Oct 30 19:47:17.668216 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:17.668186 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" path="/var/lib/kubelet/pods/b0370e1f-45da-4c11-b695-80c2db8aca48/volumes" Oct 30 19:47:18.195986 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:18.195949 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:47:18.315838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:18.315796 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" event={"ID":"da61215e-da45-4fd6-88b9-892e00d91696","Type":"ContainerStarted","Data":"0b2d322e1c03a48b76cbbfa8e66cbf1870060a24f9f96059014134d06a7395fe"} Oct 30 19:47:18.316032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:18.315852 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" event={"ID":"da61215e-da45-4fd6-88b9-892e00d91696","Type":"ContainerStarted","Data":"c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb"} Oct 30 19:47:18.427023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:18.426978 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" podStartSLOduration=2.426963931 podStartE2EDuration="2.426963931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:18.426260148 +0000 UTC m=+3674.928170639" watchObservedRunningTime="2026-10-30 19:47:18.426963931 +0000 UTC m=+3674.928874417" Oct 30 19:47:19.325535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:19.325501 4789 generic.go:356] "Generic (PLEG): container finished" podID="da61215e-da45-4fd6-88b9-892e00d91696" containerID="0b2d322e1c03a48b76cbbfa8e66cbf1870060a24f9f96059014134d06a7395fe" exitCode=0 Oct 30 19:47:19.325960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:19.325561 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" event={"ID":"da61215e-da45-4fd6-88b9-892e00d91696","Type":"ContainerDied","Data":"0b2d322e1c03a48b76cbbfa8e66cbf1870060a24f9f96059014134d06a7395fe"} Oct 30 19:47:20.490431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.490403 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649"] Oct 30 19:47:20.491149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.491128 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" Oct 30 19:47:20.491290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.491152 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" Oct 30 19:47:20.491362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.491352 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0370e1f-45da-4c11-b695-80c2db8aca48" containerName="istio-proxy" Oct 30 19:47:20.492199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.492177 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:20.494558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.494540 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.527697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.527669 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\") pod \"da61215e-da45-4fd6-88b9-892e00d91696\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " Oct 30 19:47:20.527801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.527786 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\") pod \"da61215e-da45-4fd6-88b9-892e00d91696\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " Oct 30 19:47:20.527877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.527862 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\" (UniqueName: \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\") pod \"da61215e-da45-4fd6-88b9-892e00d91696\" (UID: \"da61215e-da45-4fd6-88b9-892e00d91696\") " Oct 30 19:47:20.528178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.528135 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca" pod "da61215e-da45-4fd6-88b9-892e00d91696" (UID: "da61215e-da45-4fd6-88b9-892e00d91696"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:20.528178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.528157 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.528380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.528301 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.528448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.528425 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djd2\" (UniqueName: \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.528600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.528582 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:20.529871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.529844 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca" pod "da61215e-da45-4fd6-88b9-892e00d91696" (UID: "da61215e-da45-4fd6-88b9-892e00d91696"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:20.530009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.529991 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq" pod "da61215e-da45-4fd6-88b9-892e00d91696" (UID: "da61215e-da45-4fd6-88b9-892e00d91696"). InnerVolumeSpecName "kube-api-access-6j8lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:20.590485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.590425 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649"] Oct 30 19:47:20.630121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630088 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.630340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630176 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.630340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630294 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9djd2\" (UniqueName: \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.630466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630439 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da61215e-da45-4fd6-88b9-892e00d91696-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:20.630466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630456 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8lq\" (UniqueName: \"kubernetes.io/projected/da61215e-da45-4fd6-88b9-892e00d91696-kube-api-access-6j8lq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:20.630968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.630943 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.633246 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.633181 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.688456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.688430 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djd2\" (UniqueName: \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\") pod \"maas-api-revoke-sub-a19647eb-fx649\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:20.807320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:20.807285 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:21.085491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.085454 4789 patch_prober.go:29] interesting pod/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:47:21.085638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.085518 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:47:21.181721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.181696 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649"] Oct 30 19:47:21.183278 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:21.183246 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5198cc1_9605_40e6_bea1_183f2cdabdbd.slice/crio-0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d WatchSource:0}: Error finding container 0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d: Status 404 returned error can't find the container with id 0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d Oct 30 19:47:21.342666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.342631 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" event={"ID":"f5198cc1-9605-40e6-bea1-183f2cdabdbd","Type":"ContainerStarted","Data":"0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d"} Oct 30 19:47:21.344220 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.344179 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" Oct 30 19:47:21.344344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.344184 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb" event={"ID":"da61215e-da45-4fd6-88b9-892e00d91696","Type":"ContainerDied","Data":"c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb"} Oct 30 19:47:21.344344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:21.344311 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d3757540037542571f6af5ba1297d2e64b462b7abaa530b8e0ca2243d302cb" Oct 30 19:47:22.352718 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.352648 4789 generic.go:356] "Generic (PLEG): container finished" podID="f5198cc1-9605-40e6-bea1-183f2cdabdbd" containerID="ea453548df83c2ed7b6b41a0914e73aba9b0b5b950155fa2eb198ce5ea7b237b" exitCode=0 Oct 30 19:47:22.353071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.352737 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" event={"ID":"f5198cc1-9605-40e6-bea1-183f2cdabdbd","Type":"ContainerDied","Data":"ea453548df83c2ed7b6b41a0914e73aba9b0b5b950155fa2eb198ce5ea7b237b"} Oct 30 19:47:22.694205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.694179 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:47:22.754658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754634 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.754796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754708 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.754881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754862 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.754944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754925 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:22.754993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754956 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.754989 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755017 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755092 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755123 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755200 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\") pod \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\" (UID: \"fb62e1c2-f90c-498f-acda-6c0e655bf29f\") " Oct 30 19:47:22.755349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755091 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:22.755349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755325 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:22.755349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755330 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:22.755507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755474 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:22.755877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755857 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.755999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.755982 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.756069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.756008 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.756069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.756025 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.756069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.756040 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.757130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.757106 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:22.757272 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.757165 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:22.757446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.757426 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "kube-api-access-j6cbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:22.757517 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.757428 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo" pod "fb62e1c2-f90c-498f-acda-6c0e655bf29f" (UID: "fb62e1c2-f90c-498f-acda-6c0e655bf29f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:22.857825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.857764 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cbb\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-kube-api-access-j6cbb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.857825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.857797 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.857825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.857812 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:22.858037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:22.857825 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb62e1c2-f90c-498f-acda-6c0e655bf29f-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:23.359838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.359803 4789 generic.go:356] "Generic (PLEG): container finished" podID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerID="2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e" exitCode=0 Oct 30 19:47:23.360294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.359847 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" Oct 30 19:47:23.360294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.359884 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" event={"ID":"fb62e1c2-f90c-498f-acda-6c0e655bf29f","Type":"ContainerDied","Data":"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e"} Oct 30 19:47:23.360294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.359949 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4" event={"ID":"fb62e1c2-f90c-498f-acda-6c0e655bf29f","Type":"ContainerDied","Data":"cf613994ac6a6371b93e5b8b2a15ef948ca4926cd084dc6157fe75734ab52e71"} Oct 30 19:47:23.360294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.359996 4789 scope.go:122] "RemoveContainer" containerID="2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e" Oct 30 19:47:23.371286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.371266 4789 scope.go:122] "RemoveContainer" containerID="2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e" Oct 30 19:47:23.371586 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:23.371567 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e\": container with ID starting with 2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e not found: ID does not exist" containerID="2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e" Oct 30 19:47:23.371650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.371594 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e"} err="failed to get container status \"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e\": rpc error: code = NotFound desc = could not find container \"2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e\": container with ID starting with 2580b66888361b812a51aec1bee5fbe4d971909527bbdb0d940265b33295f59e not found: ID does not exist" Oct 30 19:47:23.506614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.506589 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:23.546133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.546111 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:47:23.565685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.565664 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\") pod \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " Oct 30 19:47:23.565800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.565782 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\" (UniqueName: \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\") pod \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " Oct 30 19:47:23.565901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.565885 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\") pod \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\" (UID: \"f5198cc1-9605-40e6-bea1-183f2cdabdbd\") " Oct 30 19:47:23.566273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.566249 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca" pod "f5198cc1-9605-40e6-bea1-183f2cdabdbd" (UID: "f5198cc1-9605-40e6-bea1-183f2cdabdbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:23.567796 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.567776 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca" pod "f5198cc1-9605-40e6-bea1-183f2cdabdbd" (UID: "f5198cc1-9605-40e6-bea1-183f2cdabdbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:23.567858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.567776 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2" pod "f5198cc1-9605-40e6-bea1-183f2cdabdbd" (UID: "f5198cc1-9605-40e6-bea1-183f2cdabdbd"). InnerVolumeSpecName "kube-api-access-9djd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:23.592431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.592409 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2301cb-openshift-default-7999cbc45f-r9qq4"] Oct 30 19:47:23.667075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.667010 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:23.667075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.667041 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9djd2\" (UniqueName: \"kubernetes.io/projected/f5198cc1-9605-40e6-bea1-183f2cdabdbd-kube-api-access-9djd2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:23.667075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.667056 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5198cc1-9605-40e6-bea1-183f2cdabdbd-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:23.670112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:23.670081 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" path="/var/lib/kubelet/pods/fb62e1c2-f90c-498f-acda-6c0e655bf29f/volumes" Oct 30 19:47:24.367683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:24.367656 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" Oct 30 19:47:24.367683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:24.367684 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649" event={"ID":"f5198cc1-9605-40e6-bea1-183f2cdabdbd","Type":"ContainerDied","Data":"0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d"} Oct 30 19:47:24.368196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:24.367734 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef0f9f90624a4d59bbacb7547561706ce3f3da854af285ebc58809b67296f9d" Oct 30 19:47:30.396585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:30.396550 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:47:31.771239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:31.771192 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:47:32.011992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.011957 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-e1a425"] Oct 30 19:47:32.012696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012669 4789 cpu_manager.go:414] "containerMap: removing container" podUID="da61215e-da45-4fd6-88b9-892e00d91696" containerName="revoke-keys" Oct 30 19:47:32.012696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012695 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da61215e-da45-4fd6-88b9-892e00d91696" containerName="revoke-keys" Oct 30 19:47:32.012894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012745 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f5198cc1-9605-40e6-bea1-183f2cdabdbd" containerName="revoke-keys" Oct 30 19:47:32.012894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012753 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5198cc1-9605-40e6-bea1-183f2cdabdbd" containerName="revoke-keys" Oct 30 19:47:32.012894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012771 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" Oct 30 19:47:32.012894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012779 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" Oct 30 19:47:32.013112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012953 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5198cc1-9605-40e6-bea1-183f2cdabdbd" containerName="revoke-keys" Oct 30 19:47:32.013112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012969 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb62e1c2-f90c-498f-acda-6c0e655bf29f" containerName="istio-proxy" Oct 30 19:47:32.013112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.012986 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="da61215e-da45-4fd6-88b9-892e00d91696" containerName="revoke-keys" Oct 30 19:47:32.016842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.016820 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:32.066123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.066085 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7hnn9\"" Oct 30 19:47:32.068576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.068551 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-e1a425"] Oct 30 19:47:32.157267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.157227 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpqn\" (UniqueName: \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\") pod \"test-curl-7399-e1a425\" (UID: \"f5ee682c-dc56-4f16-9430-73d88e6f60b1\") " pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:32.258665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.258635 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpqn\" (UniqueName: \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\") pod \"test-curl-7399-e1a425\" (UID: \"f5ee682c-dc56-4f16-9430-73d88e6f60b1\") " pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:32.316757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.316693 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpqn\" (UniqueName: \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\") pod \"test-curl-7399-e1a425\" (UID: \"f5ee682c-dc56-4f16-9430-73d88e6f60b1\") " pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:32.329680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.329649 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:32.702819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:32.702795 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-e1a425"] Oct 30 19:47:32.705237 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:32.705177 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5ee682c_dc56_4f16_9430_73d88e6f60b1.slice/crio-cf5473b902f9b65b6535fd5a7e27bf6e10f4d81e741211531afb77364bfeed53 WatchSource:0}: Error finding container cf5473b902f9b65b6535fd5a7e27bf6e10f4d81e741211531afb77364bfeed53: Status 404 returned error can't find the container with id cf5473b902f9b65b6535fd5a7e27bf6e10f4d81e741211531afb77364bfeed53 Oct 30 19:47:33.432462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:33.432427 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-e1a425" event={"ID":"f5ee682c-dc56-4f16-9430-73d88e6f60b1","Type":"ContainerStarted","Data":"4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019"} Oct 30 19:47:33.432916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:33.432469 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-e1a425" event={"ID":"f5ee682c-dc56-4f16-9430-73d88e6f60b1","Type":"ContainerStarted","Data":"cf5473b902f9b65b6535fd5a7e27bf6e10f4d81e741211531afb77364bfeed53"} Oct 30 19:47:33.541915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:33.541864 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-e1a425" podStartSLOduration=2.541849817 podStartE2EDuration="2.541849817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:33.539380729 +0000 UTC m=+3690.041291216" watchObservedRunningTime="2026-10-30 19:47:33.541849817 +0000 UTC m=+3690.043760302" Oct 30 19:47:35.002426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.002380 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-e1a425"] Oct 30 19:47:35.009934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.009905 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-e1a425"] Oct 30 19:47:35.166014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.165980 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-2ba23f"] Oct 30 19:47:35.166653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.166633 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" containerName="test-curl-7399-e1a425" Oct 30 19:47:35.166752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.166654 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" containerName="test-curl-7399-e1a425" Oct 30 19:47:35.166752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.166740 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" containerName="test-curl-7399-e1a425" Oct 30 19:47:35.170222 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.170192 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:35.220857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.220819 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-2ba23f"] Oct 30 19:47:35.288526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.288439 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ckj\" (UniqueName: \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\") pod \"test-curl-7399-2ba23f\" (UID: \"4c73497d-ee7b-4983-b042-11bf04346991\") " pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:35.389707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.389673 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ckj\" (UniqueName: \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\") pod \"test-curl-7399-2ba23f\" (UID: \"4c73497d-ee7b-4983-b042-11bf04346991\") " pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:35.444677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.444636 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-e1a425" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" containerName="test-curl-7399-e1a425" containerID="cri-o://4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019" gracePeriod=2 Oct 30 19:47:35.447883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.447855 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ckj\" (UniqueName: \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\") pod \"test-curl-7399-2ba23f\" (UID: \"4c73497d-ee7b-4983-b042-11bf04346991\") " pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:35.482376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.482354 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:35.492372 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:35.492340 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-e1a425\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:35.853589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:35.853564 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-2ba23f"] Oct 30 19:47:35.855836 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:35.855803 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c73497d_ee7b_4983_b042_11bf04346991.slice/crio-ec01cb7f1d87bea11e908342dec737ed202eaa96ede9bebe7bed1e785dba7759 WatchSource:0}: Error finding container ec01cb7f1d87bea11e908342dec737ed202eaa96ede9bebe7bed1e785dba7759: Status 404 returned error can't find the container with id ec01cb7f1d87bea11e908342dec737ed202eaa96ede9bebe7bed1e785dba7759 Oct 30 19:47:36.453665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:36.453628 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-2ba23f" event={"ID":"4c73497d-ee7b-4983-b042-11bf04346991","Type":"ContainerStarted","Data":"55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a"} Oct 30 19:47:36.453665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:36.453675 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-2ba23f" event={"ID":"4c73497d-ee7b-4983-b042-11bf04346991","Type":"ContainerStarted","Data":"ec01cb7f1d87bea11e908342dec737ed202eaa96ede9bebe7bed1e785dba7759"} Oct 30 19:47:36.561047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:36.560994 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-2ba23f" podStartSLOduration=1.5609817270000002 podStartE2EDuration="1.560981727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:36.560365768 +0000 UTC m=+3693.062276255" watchObservedRunningTime="2026-10-30 19:47:36.560981727 +0000 UTC m=+3693.062892213" Oct 30 19:47:37.672897 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:37.672867 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:37.818488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:37.818401 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\" (UniqueName: \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\") pod \"f5ee682c-dc56-4f16-9430-73d88e6f60b1\" (UID: \"f5ee682c-dc56-4f16-9430-73d88e6f60b1\") " Oct 30 19:47:37.820592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:37.820560 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn" pod "f5ee682c-dc56-4f16-9430-73d88e6f60b1" (UID: "f5ee682c-dc56-4f16-9430-73d88e6f60b1"). InnerVolumeSpecName "kube-api-access-jvpqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:37.920093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:37.920057 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpqn\" (UniqueName: \"kubernetes.io/projected/f5ee682c-dc56-4f16-9430-73d88e6f60b1-kube-api-access-jvpqn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:38.099111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.099018 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-2ba23f"] Oct 30 19:47:38.104585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.104558 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-2ba23f"] Oct 30 19:47:38.471813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.471777 4789 generic.go:356] "Generic (PLEG): container finished" podID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" containerID="4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019" exitCode=137 Oct 30 19:47:38.471997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.471824 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:38.471997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.471856 4789 scope.go:122] "RemoveContainer" containerID="4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019" Oct 30 19:47:38.472128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.472076 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-2ba23f" podUID="4c73497d-ee7b-4983-b042-11bf04346991" containerName="test-curl-7399-2ba23f" containerID="cri-o://55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a" gracePeriod=2 Oct 30 19:47:38.482396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.482377 4789 scope.go:122] "RemoveContainer" containerID="4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019" Oct 30 19:47:38.482684 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.482663 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019\": container with ID starting with 4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019 not found: ID does not exist" containerID="4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019" Oct 30 19:47:38.482745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.482694 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019"} err="failed to get container status \"4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019\": rpc error: code = NotFound desc = could not find container \"4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019\": container with ID starting with 4ad2d6869b021d28792e20bee590b260430da0d06fc88c0d7c0d22bd74aba019 not found: ID does not exist" Oct 30 19:47:38.519737 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.519700 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-2ba23f\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="4c73497d-ee7b-4983-b042-11bf04346991" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:38.534919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.534890 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-71939c"] Oct 30 19:47:38.535755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.535729 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4c73497d-ee7b-4983-b042-11bf04346991" containerName="test-curl-7399-2ba23f" Oct 30 19:47:38.535755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.535754 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c73497d-ee7b-4983-b042-11bf04346991" containerName="test-curl-7399-2ba23f" Oct 30 19:47:38.535931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.535917 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c73497d-ee7b-4983-b042-11bf04346991" containerName="test-curl-7399-2ba23f" Oct 30 19:47:38.541096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.541075 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:38.567425 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.567385 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-e1a425\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:38.589409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.589384 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-71939c"] Oct 30 19:47:38.614966 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.614927 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-2ba23f\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="4c73497d-ee7b-4983-b042-11bf04346991" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:38.662717 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.662685 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-e1a425\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" pod="opendatahub/test-curl-7399-e1a425" Oct 30 19:47:38.710615 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:38.710574 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-2ba23f\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="4c73497d-ee7b-4983-b042-11bf04346991" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:38.729472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.729376 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksqxl\" (UniqueName: \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\") pod \"test-curl-7399-71939c\" (UID: \"11603ccb-afe5-4f29-93c2-e3541f7961fb\") " pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:38.830807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.830762 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksqxl\" (UniqueName: \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\") pod \"test-curl-7399-71939c\" (UID: \"11603ccb-afe5-4f29-93c2-e3541f7961fb\") " pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:38.888808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:38.888774 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksqxl\" (UniqueName: \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\") pod \"test-curl-7399-71939c\" (UID: \"11603ccb-afe5-4f29-93c2-e3541f7961fb\") " pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:39.153651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:39.153557 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:39.539626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:39.539592 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-71939c"] Oct 30 19:47:39.540071 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:39.540038 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11603ccb_afe5_4f29_93c2_e3541f7961fb.slice/crio-fe169ee0bb574523e273176a54b42def9909bf5d6dc8928c775859c506b0bd2d WatchSource:0}: Error finding container fe169ee0bb574523e273176a54b42def9909bf5d6dc8928c775859c506b0bd2d: Status 404 returned error can't find the container with id fe169ee0bb574523e273176a54b42def9909bf5d6dc8928c775859c506b0bd2d Oct 30 19:47:39.668768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:39.668741 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ee682c-dc56-4f16-9430-73d88e6f60b1" path="/var/lib/kubelet/pods/f5ee682c-dc56-4f16-9430-73d88e6f60b1/volumes" Oct 30 19:47:40.487276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.487244 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-71939c" event={"ID":"11603ccb-afe5-4f29-93c2-e3541f7961fb","Type":"ContainerStarted","Data":"92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2"} Oct 30 19:47:40.487619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.487291 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-71939c" event={"ID":"11603ccb-afe5-4f29-93c2-e3541f7961fb","Type":"ContainerStarted","Data":"fe169ee0bb574523e273176a54b42def9909bf5d6dc8928c775859c506b0bd2d"} Oct 30 19:47:40.595178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.594842 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-71939c" podStartSLOduration=2.594825727 podStartE2EDuration="2.594825727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:40.594811085 +0000 UTC m=+3697.096721573" watchObservedRunningTime="2026-10-30 19:47:40.594825727 +0000 UTC m=+3697.096736214" Oct 30 19:47:40.713353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.713327 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:40.853787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.853715 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\" (UniqueName: \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\") pod \"4c73497d-ee7b-4983-b042-11bf04346991\" (UID: \"4c73497d-ee7b-4983-b042-11bf04346991\") " Oct 30 19:47:40.855741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.855717 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj" pod "4c73497d-ee7b-4983-b042-11bf04346991" (UID: "4c73497d-ee7b-4983-b042-11bf04346991"). InnerVolumeSpecName "kube-api-access-f7ckj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:40.955205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:40.955176 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ckj\" (UniqueName: \"kubernetes.io/projected/4c73497d-ee7b-4983-b042-11bf04346991-kube-api-access-f7ckj\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:41.496070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.496027 4789 generic.go:356] "Generic (PLEG): container finished" podID="4c73497d-ee7b-4983-b042-11bf04346991" containerID="55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a" exitCode=137 Oct 30 19:47:41.496545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.496077 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:41.496545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.496123 4789 scope.go:122] "RemoveContainer" containerID="55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a" Oct 30 19:47:41.506708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.506689 4789 scope.go:122] "RemoveContainer" containerID="55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a" Oct 30 19:47:41.507014 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:41.506992 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a\": container with ID starting with 55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a not found: ID does not exist" containerID="55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a" Oct 30 19:47:41.507095 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.507026 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a"} err="failed to get container status \"55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a\": rpc error: code = NotFound desc = could not find container \"55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a\": container with ID starting with 55707be3e3e0bd9c0fb0753bbc0745ebe7fe65af22f86cf6ec074b7a869c4c1a not found: ID does not exist" Oct 30 19:47:41.553464 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:41.553432 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-2ba23f\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="4c73497d-ee7b-4983-b042-11bf04346991" pod="opendatahub/test-curl-7399-2ba23f" Oct 30 19:47:41.668109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:41.668083 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c73497d-ee7b-4983-b042-11bf04346991" path="/var/lib/kubelet/pods/4c73497d-ee7b-4983-b042-11bf04346991/volumes" Oct 30 19:47:42.047541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.047512 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-71939c"] Oct 30 19:47:42.053491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.053462 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-71939c"] Oct 30 19:47:42.198905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.198870 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-5b53a3"] Oct 30 19:47:42.199729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.199705 4789 cpu_manager.go:414] "containerMap: removing container" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" containerName="test-curl-7399-71939c" Oct 30 19:47:42.199778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.199732 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" containerName="test-curl-7399-71939c" Oct 30 19:47:42.199889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.199877 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" containerName="test-curl-7399-71939c" Oct 30 19:47:42.205124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.205104 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:42.256375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.256349 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-5b53a3"] Oct 30 19:47:42.265884 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.265861 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8n65\" (UniqueName: \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\") pod \"test-curl-7399-5b53a3\" (UID: \"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a\") " pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:42.367232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.367135 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8n65\" (UniqueName: \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\") pod \"test-curl-7399-5b53a3\" (UID: \"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a\") " pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:42.441746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.441715 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8n65\" (UniqueName: \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\") pod \"test-curl-7399-5b53a3\" (UID: \"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a\") " pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:42.503094 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.503064 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-71939c" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" containerName="test-curl-7399-71939c" containerID="cri-o://92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2" gracePeriod=2 Oct 30 19:47:42.518302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.518276 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:42.551024 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:42.550988 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-71939c\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:42.899480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:42.899449 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-5b53a3"] Oct 30 19:47:42.901819 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:42.901774 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32a217c8_18a3_486a_ad9c_1ec10fcd4d5a.slice/crio-097226878cb20033fb60e00329dcab6b5ee5fe443614640032bb7a20e1cc2003 WatchSource:0}: Error finding container 097226878cb20033fb60e00329dcab6b5ee5fe443614640032bb7a20e1cc2003: Status 404 returned error can't find the container with id 097226878cb20033fb60e00329dcab6b5ee5fe443614640032bb7a20e1cc2003 Oct 30 19:47:43.512575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:43.512531 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-5b53a3" event={"ID":"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a","Type":"ContainerStarted","Data":"331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d"} Oct 30 19:47:43.513045 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:43.512585 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-5b53a3" event={"ID":"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a","Type":"ContainerStarted","Data":"097226878cb20033fb60e00329dcab6b5ee5fe443614640032bb7a20e1cc2003"} Oct 30 19:47:43.560274 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:43.560242 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-71939c\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:43.672670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:43.672607 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-5b53a3" podStartSLOduration=1.672591078 podStartE2EDuration="1.672591078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:43.669912317 +0000 UTC m=+3700.171822802" watchObservedRunningTime="2026-10-30 19:47:43.672591078 +0000 UTC m=+3700.174501566" Oct 30 19:47:44.761652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:44.761596 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:44.794286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:44.794264 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\" (UniqueName: \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\") pod \"11603ccb-afe5-4f29-93c2-e3541f7961fb\" (UID: \"11603ccb-afe5-4f29-93c2-e3541f7961fb\") " Oct 30 19:47:44.796326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:44.796298 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl" pod "11603ccb-afe5-4f29-93c2-e3541f7961fb" (UID: "11603ccb-afe5-4f29-93c2-e3541f7961fb"). InnerVolumeSpecName "kube-api-access-ksqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:44.809579 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:44.809548 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-71939c\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:44.895615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:44.895584 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksqxl\" (UniqueName: \"kubernetes.io/projected/11603ccb-afe5-4f29-93c2-e3541f7961fb-kube-api-access-ksqxl\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.271828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.271797 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-5b53a3"] Oct 30 19:47:45.276242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.276203 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-5b53a3"] Oct 30 19:47:45.417356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.417323 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" podUID="f3db2e92-703f-43ff-be8b-a32ef4310498" containerName="main" containerID="cri-o://6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95" gracePeriod=45 Oct 30 19:47:45.433788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.433760 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-ee9881"] Oct 30 19:47:45.434317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.434300 4789 cpu_manager.go:414] "containerMap: removing container" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" containerName="test-curl-7399-5b53a3" Oct 30 19:47:45.434317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.434316 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" containerName="test-curl-7399-5b53a3" Oct 30 19:47:45.434550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.434425 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" containerName="test-curl-7399-5b53a3" Oct 30 19:47:45.437905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.437798 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:45.485576 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:45.485546 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-71939c\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:45.503688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.503666 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8q5\" (UniqueName: \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\") pod \"test-curl-7399-ee9881\" (UID: \"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34\") " pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:45.508386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.508366 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-ee9881"] Oct 30 19:47:45.531191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.531131 4789 generic.go:356] "Generic (PLEG): container finished" podID="11603ccb-afe5-4f29-93c2-e3541f7961fb" containerID="92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2" exitCode=137 Oct 30 19:47:45.531342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.531186 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:45.531478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.531455 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-5b53a3" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" containerName="test-curl-7399-5b53a3" containerID="cri-o://331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d" gracePeriod=2 Oct 30 19:47:45.531602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.531586 4789 scope.go:122] "RemoveContainer" containerID="92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2" Oct 30 19:47:45.543521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.543501 4789 scope.go:122] "RemoveContainer" containerID="92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2" Oct 30 19:47:45.543823 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:45.543798 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2\": container with ID starting with 92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2 not found: ID does not exist" containerID="92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2" Oct 30 19:47:45.543892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.543834 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2"} err="failed to get container status \"92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2\": rpc error: code = NotFound desc = could not find container \"92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2\": container with ID starting with 92d5a991a4cb62b8f011be3621be7fcccb849cdd03a8aac7eee83b548d2c68d2 not found: ID does not exist" Oct 30 19:47:45.604965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.604934 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8q5\" (UniqueName: \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\") pod \"test-curl-7399-ee9881\" (UID: \"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34\") " pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:45.669547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.669514 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" path="/var/lib/kubelet/pods/11603ccb-afe5-4f29-93c2-e3541f7961fb/volumes" Oct 30 19:47:45.674150 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.674126 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8q5\" (UniqueName: \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\") pod \"test-curl-7399-ee9881\" (UID: \"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34\") " pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:45.680680 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:45.680653 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-71939c\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="11603ccb-afe5-4f29-93c2-e3541f7961fb" pod="opendatahub/test-curl-7399-71939c" Oct 30 19:47:45.701186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.701165 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:47:45.728264 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:45.728234 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-5b53a3\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:45.798461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.798404 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:45.807520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807498 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.807600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807565 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.807646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807597 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.807646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807630 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\" (UniqueName: \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.807729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807699 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.807784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807733 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:45.807838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807824 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\") pod \"f3db2e92-703f-43ff-be8b-a32ef4310498\" (UID: \"f3db2e92-703f-43ff-be8b-a32ef4310498\") " Oct 30 19:47:45.808157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.807893 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:45.808157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.808030 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:45.808347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.808305 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.808347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.808324 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.808347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.808340 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.810108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.810082 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:45.810289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.810116 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "kube-api-access-78986". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:45.810289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.810135 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm" pod "f3db2e92-703f-43ff-be8b-a32ef4310498" (UID: "f3db2e92-703f-43ff-be8b-a32ef4310498"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:45.914630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.914587 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3db2e92-703f-43ff-be8b-a32ef4310498-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.914630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.914621 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3db2e92-703f-43ff-be8b-a32ef4310498-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:45.914630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:45.914636 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78986\" (UniqueName: \"kubernetes.io/projected/f3db2e92-703f-43ff-be8b-a32ef4310498-kube-api-access-78986\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:46.175408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.175384 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-ee9881"] Oct 30 19:47:46.177375 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:46.177351 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e2ae4b_c9ec_4d78_93c2_d5cec01d0a34.slice/crio-91cad3673c0d0b4ad3a8adeee18a41cfa35e13993cc06fe812baf4632399dfd2 WatchSource:0}: Error finding container 91cad3673c0d0b4ad3a8adeee18a41cfa35e13993cc06fe812baf4632399dfd2: Status 404 returned error can't find the container with id 91cad3673c0d0b4ad3a8adeee18a41cfa35e13993cc06fe812baf4632399dfd2 Oct 30 19:47:46.539293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.539258 4789 generic.go:356] "Generic (PLEG): container finished" podID="f3db2e92-703f-43ff-be8b-a32ef4310498" containerID="6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95" exitCode=0 Oct 30 19:47:46.539513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.539317 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" Oct 30 19:47:46.539513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.539317 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" event={"ID":"f3db2e92-703f-43ff-be8b-a32ef4310498","Type":"ContainerDied","Data":"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95"} Oct 30 19:47:46.539513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.539365 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm" event={"ID":"f3db2e92-703f-43ff-be8b-a32ef4310498","Type":"ContainerDied","Data":"895463c20c6fceee2fc53fd5ec0bfc75a94ab4fdcccfb1f42e728c671c0f7516"} Oct 30 19:47:46.539513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.539385 4789 scope.go:122] "RemoveContainer" containerID="6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95" Oct 30 19:47:46.542728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.542699 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-ee9881" event={"ID":"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34","Type":"ContainerStarted","Data":"590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3"} Oct 30 19:47:46.542880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.542743 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-ee9881" event={"ID":"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34","Type":"ContainerStarted","Data":"91cad3673c0d0b4ad3a8adeee18a41cfa35e13993cc06fe812baf4632399dfd2"} Oct 30 19:47:46.549652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.549618 4789 scope.go:122] "RemoveContainer" containerID="6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95" Oct 30 19:47:46.549928 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:46.549904 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95\": container with ID starting with 6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95 not found: ID does not exist" containerID="6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95" Oct 30 19:47:46.550026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.549932 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95"} err="failed to get container status \"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95\": rpc error: code = NotFound desc = could not find container \"6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95\": container with ID starting with 6bc3b69c1b2acc14e5ae14f3c4f6a0b75fc74a944df18af99b7ff0216ea28f95 not found: ID does not exist" Oct 30 19:47:46.702459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.702420 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:47:46.706011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.705986 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-db06cf/auth-test-model-db06cf-kserve-6cc4f4844f-gcthm"] Oct 30 19:47:46.791961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.791850 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" podUID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" containerName="main" containerID="cri-o://ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c" gracePeriod=45 Oct 30 19:47:46.820063 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:46.819862 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-ee9881" podStartSLOduration=1.81984391 podStartE2EDuration="1.81984391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:46.819513469 +0000 UTC m=+3703.321423958" watchObservedRunningTime="2026-10-30 19:47:46.81984391 +0000 UTC m=+3703.321754397" Oct 30 19:47:47.064809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.064775 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:47:47.129766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129738 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\" (UniqueName: \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.129905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129804 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.129905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129822 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.129905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129852 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.129905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129873 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.130103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.129933 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\") pod \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\" (UID: \"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6\") " Oct 30 19:47:47.130165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.130145 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:47.130263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.130156 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:47.130366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.130335 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:47.130492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.130404 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.130492 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.130451 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.132256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.132205 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:47.132256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.132243 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:47.132393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.132318 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg" pod "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" (UID: "08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6"). InnerVolumeSpecName "kube-api-access-qp5dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:47.231503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.231476 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.231503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.231502 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5dg\" (UniqueName: \"kubernetes.io/projected/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-kube-api-access-qp5dg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.231678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.231514 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.231678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.231523 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:47.550974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.550944 4789 generic.go:356] "Generic (PLEG): container finished" podID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" containerID="ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c" exitCode=0 Oct 30 19:47:47.551089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.550990 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" Oct 30 19:47:47.551089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.551004 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" event={"ID":"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6","Type":"ContainerDied","Data":"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c"} Oct 30 19:47:47.551089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.551049 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548" event={"ID":"08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6","Type":"ContainerDied","Data":"d4d8f8ab3c0ade250e0e99db8b30452ab2d7d5c9b36ed5348406aac4eaae834b"} Oct 30 19:47:47.551089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.551069 4789 scope.go:122] "RemoveContainer" containerID="ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c" Oct 30 19:47:47.591562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.591536 4789 scope.go:122] "RemoveContainer" containerID="ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c" Oct 30 19:47:47.591883 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:47.591855 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c\": container with ID starting with ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c not found: ID does not exist" containerID="ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c" Oct 30 19:47:47.591992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.591889 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c"} err="failed to get container status \"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c\": rpc error: code = NotFound desc = could not find container \"ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c\": container with ID starting with ce853b9ef07cd4226b0c1c1c0cc33c27bb35d86074e7652262522073d145808c not found: ID does not exist" Oct 30 19:47:47.667791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.667761 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3db2e92-703f-43ff-be8b-a32ef4310498" path="/var/lib/kubelet/pods/f3db2e92-703f-43ff-be8b-a32ef4310498/volumes" Oct 30 19:47:47.710333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.710288 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:47:47.711669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.711648 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b2604c/auth-test-model-b2604c-kserve-f5f8dbc7-tq548"] Oct 30 19:47:47.742422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.742401 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:47.838445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.838373 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\" (UniqueName: \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\") pod \"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a\" (UID: \"32a217c8-18a3-486a-ad9c-1ec10fcd4d5a\") " Oct 30 19:47:47.840397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.840376 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65" pod "32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" (UID: "32a217c8-18a3-486a-ad9c-1ec10fcd4d5a"). InnerVolumeSpecName "kube-api-access-n8n65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:47.940104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:47.940068 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8n65\" (UniqueName: \"kubernetes.io/projected/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a-kube-api-access-n8n65\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:48.196173 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:48.196140 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:47:48.313582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.313556 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-ee9881"] Oct 30 19:47:48.317939 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.317914 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-ee9881"] Oct 30 19:47:48.473065 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.472988 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7399-135a7e"] Oct 30 19:47:48.473636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473616 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f3db2e92-703f-43ff-be8b-a32ef4310498" containerName="main" Oct 30 19:47:48.473738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473637 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3db2e92-703f-43ff-be8b-a32ef4310498" containerName="main" Oct 30 19:47:48.473738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473652 4789 cpu_manager.go:414] "containerMap: removing container" podUID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" containerName="main" Oct 30 19:47:48.473738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473661 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" containerName="main" Oct 30 19:47:48.473738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473689 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" containerName="test-curl-7399-ee9881" Oct 30 19:47:48.473738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473697 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" containerName="test-curl-7399-ee9881" Oct 30 19:47:48.474014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473834 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3db2e92-703f-43ff-be8b-a32ef4310498" containerName="main" Oct 30 19:47:48.474014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473849 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" containerName="main" Oct 30 19:47:48.474014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.473863 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" containerName="test-curl-7399-ee9881" Oct 30 19:47:48.477027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.477009 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:48.529980 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.529957 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-135a7e"] Oct 30 19:47:48.546636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.546615 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-425qr\" (UniqueName: \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\") pod \"test-curl-7399-135a7e\" (UID: \"3bb64c06-a380-44a8-9687-b7ffc6d243c5\") " pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:48.560093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.560065 4789 generic.go:356] "Generic (PLEG): container finished" podID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" containerID="331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d" exitCode=137 Oct 30 19:47:48.560185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.560127 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:48.560185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.560164 4789 scope.go:122] "RemoveContainer" containerID="331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d" Oct 30 19:47:48.561563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.561445 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-ee9881" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" containerName="test-curl-7399-ee9881" containerID="cri-o://590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3" gracePeriod=2 Oct 30 19:47:48.570004 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.569986 4789 scope.go:122] "RemoveContainer" containerID="331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d" Oct 30 19:47:48.570442 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:48.570418 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d\": container with ID starting with 331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d not found: ID does not exist" containerID="331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d" Oct 30 19:47:48.570528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.570454 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d"} err="failed to get container status \"331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d\": rpc error: code = NotFound desc = could not find container \"331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d\": container with ID starting with 331e4fb043204d6935d5ec62419ce0f32f417b3ab4a1a9a360729c7511e7404d not found: ID does not exist" Oct 30 19:47:48.632775 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:48.632747 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-ee9881\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:48.647294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.647270 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-425qr\" (UniqueName: \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\") pod \"test-curl-7399-135a7e\" (UID: \"3bb64c06-a380-44a8-9687-b7ffc6d243c5\") " pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:48.680407 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:48.680381 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-5b53a3\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:48.704931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.704903 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-425qr\" (UniqueName: \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\") pod \"test-curl-7399-135a7e\" (UID: \"3bb64c06-a380-44a8-9687-b7ffc6d243c5\") " pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:48.789565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:48.789505 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:49.161712 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.161683 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7399-135a7e"] Oct 30 19:47:49.163599 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:49.163566 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bb64c06_a380_44a8_9687_b7ffc6d243c5.slice/crio-b6f5b17e5845545d956065e8c465fa0ee201795a75dcc22029776e87b77b0ef0 WatchSource:0}: Error finding container b6f5b17e5845545d956065e8c465fa0ee201795a75dcc22029776e87b77b0ef0: Status 404 returned error can't find the container with id b6f5b17e5845545d956065e8c465fa0ee201795a75dcc22029776e87b77b0ef0 Oct 30 19:47:49.572653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.572617 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-135a7e" event={"ID":"3bb64c06-a380-44a8-9687-b7ffc6d243c5","Type":"ContainerStarted","Data":"630235e42845ae7a8e65485cca2abbecc0248b14e3ab6ab4fee750ca02b2b78b"} Oct 30 19:47:49.572653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.572659 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7399-135a7e" event={"ID":"3bb64c06-a380-44a8-9687-b7ffc6d243c5","Type":"ContainerStarted","Data":"b6f5b17e5845545d956065e8c465fa0ee201795a75dcc22029776e87b77b0ef0"} Oct 30 19:47:49.668667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.668626 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6" path="/var/lib/kubelet/pods/08c6825a-2fb4-45b1-b7ff-6e3c91dc50d6/volumes" Oct 30 19:47:49.669071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.669048 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" path="/var/lib/kubelet/pods/32a217c8-18a3-486a-ad9c-1ec10fcd4d5a/volumes" Oct 30 19:47:49.683020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:49.682976 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7399-135a7e" podStartSLOduration=1.682963574 podStartE2EDuration="1.682963574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:49.681841973 +0000 UTC m=+3706.183752462" watchObservedRunningTime="2026-10-30 19:47:49.682963574 +0000 UTC m=+3706.184874054" Oct 30 19:47:49.729523 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:49.729479 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-5b53a3\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="32a217c8-18a3-486a-ad9c-1ec10fcd4d5a" pod="opendatahub/test-curl-7399-5b53a3" Oct 30 19:47:49.777417 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:49.777378 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-ee9881\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:50.813941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:50.813914 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:50.879337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:50.879267 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\" (UniqueName: \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\") pod \"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34\" (UID: \"e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34\") " Oct 30 19:47:50.881155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:50.881125 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5" pod "e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" (UID: "e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34"). InnerVolumeSpecName "kube-api-access-qv8q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:50.980297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:50.980266 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8q5\" (UniqueName: \"kubernetes.io/projected/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34-kube-api-access-qv8q5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:51.162677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.162587 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7399-135a7e"] Oct 30 19:47:51.169087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.169059 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7399-135a7e"] Oct 30 19:47:51.587716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.587683 4789 generic.go:356] "Generic (PLEG): container finished" podID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" containerID="590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3" exitCode=137 Oct 30 19:47:51.587865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.587733 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:51.587865 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.587772 4789 scope.go:122] "RemoveContainer" containerID="590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3" Oct 30 19:47:51.588077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.588053 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7399-135a7e" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" containerName="test-curl-7399-135a7e" containerID="cri-o://630235e42845ae7a8e65485cca2abbecc0248b14e3ab6ab4fee750ca02b2b78b" gracePeriod=2 Oct 30 19:47:51.597070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.597054 4789 scope.go:122] "RemoveContainer" containerID="590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3" Oct 30 19:47:51.597365 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:51.597345 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3\": container with ID starting with 590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3 not found: ID does not exist" containerID="590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3" Oct 30 19:47:51.597439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.597373 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3"} err="failed to get container status \"590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3\": rpc error: code = NotFound desc = could not find container \"590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3\": container with ID starting with 590c017a97661095c2436a9fac1fc7cff732732504d7602538b1740a585057d3 not found: ID does not exist" Oct 30 19:47:51.635937 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:51.635907 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:51.669247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:51.669220 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" path="/var/lib/kubelet/pods/e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34/volumes" Oct 30 19:47:51.683609 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:51.683571 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:51.731033 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:51.730998 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-ee9881\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="e0e2ae4b-c9ec-4d78-93c2-d5cec01d0a34" pod="opendatahub/test-curl-7399-ee9881" Oct 30 19:47:53.241542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.241497 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:47:53.242039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.242024 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" containerName="test-curl-7399-135a7e" Oct 30 19:47:53.242081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.242039 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" containerName="test-curl-7399-135a7e" Oct 30 19:47:53.242147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.242138 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" containerName="test-curl-7399-135a7e" Oct 30 19:47:53.246575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.246555 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.298326 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.297700 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-17216f-openshift-default-dockercfg-8wshj\"" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304040 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304091 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304143 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304179 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304280 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304308 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304343 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304382 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.305330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.304490 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqj6k\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.348393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.348362 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:47:53.405687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.405656 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqj6k\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.405926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.405716 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.405926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.405750 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.405926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.405797 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406056 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406139 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406168 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406224 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406270 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406308 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406323 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406655 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.406797 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.406707 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.407076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.407052 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.409049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.409022 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.409488 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.409469 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.463383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.463342 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqj6k\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.464097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.464062 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\") pod \"e2e-extmod-a-17216f-openshift-default-74975945-svk2t\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.560440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.560345 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:53.614041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.613949 4789 generic.go:356] "Generic (PLEG): container finished" podID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" containerID="630235e42845ae7a8e65485cca2abbecc0248b14e3ab6ab4fee750ca02b2b78b" exitCode=137 Oct 30 19:47:53.817478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.817410 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:53.865320 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:53.865277 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:53.912471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.912435 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\" (UniqueName: \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\") pod \"3bb64c06-a380-44a8-9687-b7ffc6d243c5\" (UID: \"3bb64c06-a380-44a8-9687-b7ffc6d243c5\") " Oct 30 19:47:53.914538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:53.914507 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr" pod "3bb64c06-a380-44a8-9687-b7ffc6d243c5" (UID: "3bb64c06-a380-44a8-9687-b7ffc6d243c5"). InnerVolumeSpecName "kube-api-access-425qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:54.013975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.013937 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-425qr\" (UniqueName: \"kubernetes.io/projected/3bb64c06-a380-44a8-9687-b7ffc6d243c5-kube-api-access-425qr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:54.058141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.058113 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:47:54.059866 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:47:54.059844 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98209702_5b61_426c_9470_168fa30d8f04.slice/crio-71b7fe6d111c352c718e6e25a9caab933b2ce69723a297a0ad0206e26d44c52d WatchSource:0}: Error finding container 71b7fe6d111c352c718e6e25a9caab933b2ce69723a297a0ad0206e26d44c52d: Status 404 returned error can't find the container with id 71b7fe6d111c352c718e6e25a9caab933b2ce69723a297a0ad0206e26d44c52d Oct 30 19:47:54.062558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.062528 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:54.062638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.062592 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:54.062638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.062622 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:54.624248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.622328 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:54.624248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.622364 4789 scope.go:122] "RemoveContainer" containerID="630235e42845ae7a8e65485cca2abbecc0248b14e3ab6ab4fee750ca02b2b78b" Oct 30 19:47:54.625080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.624759 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerStarted","Data":"6c160a6f7002f2d9c9393efae0ddef1a2000bc5ed306f7535b054d84056a09ab"} Oct 30 19:47:54.625080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.624807 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerStarted","Data":"71b7fe6d111c352c718e6e25a9caab933b2ce69723a297a0ad0206e26d44c52d"} Oct 30 19:47:54.670407 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:54.670368 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:54.785497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:54.785435 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podStartSLOduration=1.7854189310000002 podStartE2EDuration="1.785418931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:54.784480902 +0000 UTC m=+3711.286391389" watchObservedRunningTime="2026-10-30 19:47:54.785418931 +0000 UTC m=+3711.287329416" Oct 30 19:47:54.832290 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:54.832245 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:47:55.560702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:55.560664 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:55.566870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:55.566846 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:55.634959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:55.634928 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:55.635973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:55.635955 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:47:55.671839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:47:55.671808 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" path="/var/lib/kubelet/pods/3bb64c06-a380-44a8-9687-b7ffc6d243c5/volumes" Oct 30 19:47:55.780802 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:47:55.780731 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7399-135a7e\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="3bb64c06-a380-44a8-9687-b7ffc6d243c5" pod="opendatahub/test-curl-7399-135a7e" Oct 30 19:48:01.288794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.288757 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:48:01.294781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.294757 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.346312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.346288 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-17216f\"" Oct 30 19:48:01.352924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.352901 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:48:01.416876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.416847 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvrr\" (UniqueName: \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.417023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.416991 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.455934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.455902 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:48:01.460436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.460416 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.511678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.511650 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-17216f-dockercfg-8k6gz\"" Oct 30 19:48:01.516041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.516010 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:48:01.518728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.518699 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.518873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.518850 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvrr\" (UniqueName: \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.519775 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.519747 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.578838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.578780 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvrr\" (UniqueName: \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\") pod \"payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.608921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.608897 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:01.620518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.620495 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r24t\" (UniqueName: \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.620605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.620552 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.722413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.722379 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r24t\" (UniqueName: \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.722593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.722452 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.723143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.723123 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.780168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.780143 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r24t\" (UniqueName: \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\") pod \"payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:01.982029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:01.982003 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:48:01.983799 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:48:01.983774 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f19999_bc7b_401c_8cf1_81b65f6697dd.slice/crio-1ae5f2acf23b10ddc699bd95d794b00cd3b7b279f89378e64e7043a491800068 WatchSource:0}: Error finding container 1ae5f2acf23b10ddc699bd95d794b00cd3b7b279f89378e64e7043a491800068: Status 404 returned error can't find the container with id 1ae5f2acf23b10ddc699bd95d794b00cd3b7b279f89378e64e7043a491800068 Oct 30 19:48:02.073600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.073577 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:02.450912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.450890 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:48:02.452795 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:48:02.452767 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb28d4c9_9cd1_4d28_8393_15718836648a.slice/crio-dfc2a04ddc292be406e9f60400f252d6dff4da0b2fe5249b85371ade8cef8b8b WatchSource:0}: Error finding container dfc2a04ddc292be406e9f60400f252d6dff4da0b2fe5249b85371ade8cef8b8b: Status 404 returned error can't find the container with id dfc2a04ddc292be406e9f60400f252d6dff4da0b2fe5249b85371ade8cef8b8b Oct 30 19:48:02.696728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.696693 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" event={"ID":"db28d4c9-9cd1-4d28-8393-15718836648a","Type":"ContainerStarted","Data":"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee"} Oct 30 19:48:02.696883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.696749 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" event={"ID":"db28d4c9-9cd1-4d28-8393-15718836648a","Type":"ContainerStarted","Data":"dfc2a04ddc292be406e9f60400f252d6dff4da0b2fe5249b85371ade8cef8b8b"} Oct 30 19:48:02.696949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.696875 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:02.698340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.698311 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" event={"ID":"f4f19999-bc7b-401c-8cf1-81b65f6697dd","Type":"ContainerStarted","Data":"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e"} Oct 30 19:48:02.698443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.698353 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" event={"ID":"f4f19999-bc7b-401c-8cf1-81b65f6697dd","Type":"ContainerStarted","Data":"1ae5f2acf23b10ddc699bd95d794b00cd3b7b279f89378e64e7043a491800068"} Oct 30 19:48:02.698443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.698397 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:02.825694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.825610 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" podStartSLOduration=1.825597972 podStartE2EDuration="1.825597972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:02.822415116 +0000 UTC m=+3719.324325601" watchObservedRunningTime="2026-10-30 19:48:02.825597972 +0000 UTC m=+3719.327508486" Oct 30 19:48:02.938290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:02.938244 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" podStartSLOduration=1.938231861 podStartE2EDuration="1.938231861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:02.934889139 +0000 UTC m=+3719.436799626" watchObservedRunningTime="2026-10-30 19:48:02.938231861 +0000 UTC m=+3719.440142343" Oct 30 19:48:13.707895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:13.707824 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:48:13.707895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:13.707887 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:48:18.198370 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:48:18.198330 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:48:29.623992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.623954 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7"] Oct 30 19:48:29.634732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.634693 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.685699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.685442 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7"] Oct 30 19:48:29.685699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.685550 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:29.685699 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.685608 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:48:29.686038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.685810 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:29.686038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.685833 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:29.704096 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.704067 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.704276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.704176 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lcdz\" (UniqueName: \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.704276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.704243 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.804927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.804898 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lcdz\" (UniqueName: \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.805104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.804947 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.805104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.805026 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.805704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.805678 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.807437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.807399 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.864177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.864149 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lcdz\" (UniqueName: \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\") pod \"maas-api-revoke-sub-569b6fd1-22nv7\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:29.950743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:29.950719 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:30.329092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:30.329067 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7"] Oct 30 19:48:30.332542 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:48:30.332513 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b76897d_1c1f_4301_88a3_ed80ea6e8a99.slice/crio-2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c WatchSource:0}: Error finding container 2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c: Status 404 returned error can't find the container with id 2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c Oct 30 19:48:30.882186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:30.882112 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" event={"ID":"9b76897d-1c1f-4301-88a3-ed80ea6e8a99","Type":"ContainerStarted","Data":"7135e4f79abecda9e7ebcae12b0b1272cd5a7988f42fc5f05d7220d8d3b77d55"} Oct 30 19:48:30.882186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:30.882158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" event={"ID":"9b76897d-1c1f-4301-88a3-ed80ea6e8a99","Type":"ContainerStarted","Data":"2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c"} Oct 30 19:48:30.994708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:30.994642 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" podStartSLOduration=1.994628268 podStartE2EDuration="1.994628268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:30.99192756 +0000 UTC m=+3747.493838047" watchObservedRunningTime="2026-10-30 19:48:30.994628268 +0000 UTC m=+3747.496538754" Oct 30 19:48:31.889807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:31.889756 4789 generic.go:356] "Generic (PLEG): container finished" podID="9b76897d-1c1f-4301-88a3-ed80ea6e8a99" containerID="7135e4f79abecda9e7ebcae12b0b1272cd5a7988f42fc5f05d7220d8d3b77d55" exitCode=0 Oct 30 19:48:31.890268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:31.889956 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" event={"ID":"9b76897d-1c1f-4301-88a3-ed80ea6e8a99","Type":"ContainerDied","Data":"7135e4f79abecda9e7ebcae12b0b1272cd5a7988f42fc5f05d7220d8d3b77d55"} Oct 30 19:48:33.052393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.052369 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:33.142370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.142342 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\") pod \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " Oct 30 19:48:33.142511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.142402 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\") pod \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " Oct 30 19:48:33.142511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.142434 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\" (UniqueName: \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\") pod \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\" (UID: \"9b76897d-1c1f-4301-88a3-ed80ea6e8a99\") " Oct 30 19:48:33.142770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.142735 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca" pod "9b76897d-1c1f-4301-88a3-ed80ea6e8a99" (UID: "9b76897d-1c1f-4301-88a3-ed80ea6e8a99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:33.144311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.144282 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca" pod "9b76897d-1c1f-4301-88a3-ed80ea6e8a99" (UID: "9b76897d-1c1f-4301-88a3-ed80ea6e8a99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:33.144408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.144370 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz" pod "9b76897d-1c1f-4301-88a3-ed80ea6e8a99" (UID: "9b76897d-1c1f-4301-88a3-ed80ea6e8a99"). InnerVolumeSpecName "kube-api-access-2lcdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:33.244167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.244127 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:33.244167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.244158 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:33.244167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.244168 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lcdz\" (UniqueName: \"kubernetes.io/projected/9b76897d-1c1f-4301-88a3-ed80ea6e8a99-kube-api-access-2lcdz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:33.904036 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.904010 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" Oct 30 19:48:33.904183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.904037 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7" event={"ID":"9b76897d-1c1f-4301-88a3-ed80ea6e8a99","Type":"ContainerDied","Data":"2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c"} Oct 30 19:48:33.904183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:33.904080 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbf6d216fd77977d5c52ddeb490fd2eecf8991710586f5e697697d55f194b6c" Oct 30 19:48:38.996183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:38.996141 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:48:38.996924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:38.996902 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9b76897d-1c1f-4301-88a3-ed80ea6e8a99" containerName="revoke-keys" Oct 30 19:48:38.997022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:38.996926 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b76897d-1c1f-4301-88a3-ed80ea6e8a99" containerName="revoke-keys" Oct 30 19:48:38.997120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:38.997106 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b76897d-1c1f-4301-88a3-ed80ea6e8a99" containerName="revoke-keys" Oct 30 19:48:39.000531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.000508 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.049835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.049785 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f858d4\"/\"default-dockercfg-vc9v4\"" Oct 30 19:48:39.096187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.096156 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f858d4\"/\"autores-model-f858d4-kserve-self-signed-certs\"" Oct 30 19:48:39.096375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.096245 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f858d4\"/\"kube-root-ca.crt\"" Oct 30 19:48:39.096375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.096308 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f858d4\"/\"openshift-service-ca.crt\"" Oct 30 19:48:39.099356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.099330 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:48:39.104320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104291 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.104443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104396 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.104443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104425 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mf5m\" (UniqueName: \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.104576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104513 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.104576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104538 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.104689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.104660 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.205784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.205739 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.205986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.205797 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.205986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.205871 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.205984 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.206054 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.206081 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mf5m\" (UniqueName: \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.206191 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.206269 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.206506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.206474 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.208661 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.208640 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.208764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.208728 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.264495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.264408 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mf5m\" (UniqueName: \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\") pod \"autores-model-f858d4-kserve-6c9549b697-jr727\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.314218 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.314175 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:39.715805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.715368 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:48:39.947509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.947472 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" event={"ID":"ff47f656-2879-4cd2-827d-53aa86765bce","Type":"ContainerStarted","Data":"238840831fef6ca42b1fedf1931d926cb1d3ab2f0368aedf3b8ed22fe428d7a1"} Oct 30 19:48:39.947668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:39.947517 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" event={"ID":"ff47f656-2879-4cd2-827d-53aa86765bce","Type":"ContainerStarted","Data":"2584389081daf6cc634347143f4b08f16c47801f4911d7ec519620fc7b65adba"} Oct 30 19:48:40.061101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.061010 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" podStartSLOduration=2.060993295 podStartE2EDuration="2.060993295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:40.058232815 +0000 UTC m=+3756.560143302" watchObservedRunningTime="2026-10-30 19:48:40.060993295 +0000 UTC m=+3756.562903782" Oct 30 19:48:40.314646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.314576 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:40.352286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.352251 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7"] Oct 30 19:48:40.356413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.356391 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.409580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.409535 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:40.409746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.409675 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:48:40.410196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.410157 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:40.430781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.430750 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.430923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.430799 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.430923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.430859 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5ql\" (UniqueName: \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.447695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.447667 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:40.459670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.459648 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7"] Oct 30 19:48:40.531657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.531621 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5ql\" (UniqueName: \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.531829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.531809 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.531911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.531865 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.532723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.532693 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.534926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.534891 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.590618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.590552 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5ql\" (UniqueName: \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\") pod \"maas-api-revoke-sub-9fa8d657-m4mf7\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:40.670902 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:40.670865 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:41.072394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:41.072371 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7"] Oct 30 19:48:41.074524 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:48:41.074498 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf59d71_664d_44d1_8ff5_99c1fdb8a2f3.slice/crio-093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f WatchSource:0}: Error finding container 093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f: Status 404 returned error can't find the container with id 093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f Oct 30 19:48:41.964733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:41.964696 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" event={"ID":"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3","Type":"ContainerStarted","Data":"e134bf5e08de0f5f5aa0dcbdc637a23bfb3e9d40d8767df6179e7ccd912bb7e6"} Oct 30 19:48:41.964889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:41.964752 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" event={"ID":"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3","Type":"ContainerStarted","Data":"093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f"} Oct 30 19:48:42.075302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:42.075246 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" podStartSLOduration=2.075233664 podStartE2EDuration="2.075233664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:42.074424475 +0000 UTC m=+3758.576334962" watchObservedRunningTime="2026-10-30 19:48:42.075233664 +0000 UTC m=+3758.577144141" Oct 30 19:48:42.443350 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:42.443319 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw"] Oct 30 19:48:42.448590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:42.448562 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-af72cf-rrfpw"] Oct 30 19:48:42.973633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:42.973589 4789 generic.go:356] "Generic (PLEG): container finished" podID="bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" containerID="e134bf5e08de0f5f5aa0dcbdc637a23bfb3e9d40d8767df6179e7ccd912bb7e6" exitCode=0 Oct 30 19:48:42.973927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:42.973906 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" event={"ID":"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3","Type":"ContainerDied","Data":"e134bf5e08de0f5f5aa0dcbdc637a23bfb3e9d40d8767df6179e7ccd912bb7e6"} Oct 30 19:48:43.669972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:43.669939 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad732af7-2b5b-4354-bb73-29d403a34d83" path="/var/lib/kubelet/pods/ad732af7-2b5b-4354-bb73-29d403a34d83/volumes" Oct 30 19:48:44.153023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.152999 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:44.281039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.280967 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\") pod \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " Oct 30 19:48:44.281176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.281056 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\" (UniqueName: \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\") pod \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " Oct 30 19:48:44.281176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.281114 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\") pod \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\" (UID: \"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3\") " Oct 30 19:48:44.281341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.281293 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca" pod "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" (UID: "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:44.281623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.281596 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:44.283186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.283162 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql" pod "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" (UID: "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3"). InnerVolumeSpecName "kube-api-access-5t5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:44.283315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.283164 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca" pod "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" (UID: "bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:44.382189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.382166 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t5ql\" (UniqueName: \"kubernetes.io/projected/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-kube-api-access-5t5ql\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:44.382189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.382190 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:44.987195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.987167 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" Oct 30 19:48:44.987195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.987167 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7" event={"ID":"bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3","Type":"ContainerDied","Data":"093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f"} Oct 30 19:48:44.987795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:44.987256 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093e76e80022ae5a0e42a655421754a6266a5b63db56f182d800b3b6b268ac8f" Oct 30 19:48:47.876197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:47.876166 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:48:48.197590 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:48:48.197545 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:48:49.314547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:49.314508 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:49.331060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:49.331029 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:48:50.032583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:48:50.032546 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:49:02.898504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:02.898434 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" podUID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" containerName="main" containerID="cri-o://4b5b682da96022084e19a95ba84ba9adfb740d39367ce521179552236a14d229" gracePeriod=45 Oct 30 19:49:03.116070 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.116035 4789 generic.go:356] "Generic (PLEG): container finished" podID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" containerID="4b5b682da96022084e19a95ba84ba9adfb740d39367ce521179552236a14d229" exitCode=0 Oct 30 19:49:03.116447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.116242 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" event={"ID":"645fd7f0-95d0-4208-9383-4e2400ffb7ce","Type":"ContainerDied","Data":"4b5b682da96022084e19a95ba84ba9adfb740d39367ce521179552236a14d229"} Oct 30 19:49:03.179807 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.179784 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:49:03.278527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278502 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278645 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278678 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278728 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278767 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\" (UniqueName: \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278802 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\") pod \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\" (UID: \"645fd7f0-95d0-4208-9383-4e2400ffb7ce\") " Oct 30 19:49:03.278998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.278921 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:03.279054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.279002 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:03.279114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.279096 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:03.279351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.279297 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:03.279351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.279320 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:03.279351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.279332 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:03.280756 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.280733 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:03.280857 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.280814 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "kube-api-access-gnfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:03.280899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.280841 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs" pod "645fd7f0-95d0-4208-9383-4e2400ffb7ce" (UID: "645fd7f0-95d0-4208-9383-4e2400ffb7ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:03.380729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.380699 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/645fd7f0-95d0-4208-9383-4e2400ffb7ce-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:03.380729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.380728 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfdv\" (UniqueName: \"kubernetes.io/projected/645fd7f0-95d0-4208-9383-4e2400ffb7ce-kube-api-access-gnfdv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:03.380930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:03.380738 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/645fd7f0-95d0-4208-9383-4e2400ffb7ce-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:04.125097 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:04.125065 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" event={"ID":"645fd7f0-95d0-4208-9383-4e2400ffb7ce","Type":"ContainerDied","Data":"a97e4f9bf867fb43db1ead166c1df8b80669d10b398961f506b57e3f15ec83db"} Oct 30 19:49:04.125592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:04.125119 4789 scope.go:122] "RemoveContainer" containerID="4b5b682da96022084e19a95ba84ba9adfb740d39367ce521179552236a14d229" Oct 30 19:49:04.125592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:04.125079 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct" Oct 30 19:49:04.280957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:04.280926 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:49:04.328871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:04.328842 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-af72cf/praxis-route-af72cf-kserve-8cb896d87-r5qct"] Oct 30 19:49:05.668837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:05.668806 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" path="/var/lib/kubelet/pods/645fd7f0-95d0-4208-9383-4e2400ffb7ce/volumes" Oct 30 19:49:18.196731 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:18.196687 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:49:27.569580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:27.569543 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:49:29.642853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.642812 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:49:29.643617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643593 4789 cpu_manager.go:414] "containerMap: removing container" podUID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" containerName="main" Oct 30 19:49:29.643617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643616 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" containerName="main" Oct 30 19:49:29.643783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643629 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" containerName="revoke-keys" Oct 30 19:49:29.643783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643637 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" containerName="revoke-keys" Oct 30 19:49:29.643905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643798 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="645fd7f0-95d0-4208-9383-4e2400ffb7ce" containerName="main" Oct 30 19:49:29.643905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.643812 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" containerName="revoke-keys" Oct 30 19:49:29.648369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.648349 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.697620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.697588 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-660b59-openshift-default-dockercfg-9g48x\"" Oct 30 19:49:29.748404 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.748370 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:49:29.765271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765231 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765523 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765619 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765752 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765733 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765835 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765907 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765929 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765958 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.765989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.765987 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rhw\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.867640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867604 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rhw\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.867848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867693 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.867848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867717 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.867848 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867751 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867856 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867956 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.867988 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868015 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868054 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868141 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868430 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868484 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868612 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868599 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.868722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.868695 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.870487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.870457 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.870826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.870809 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.925803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.925735 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rhw\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.925994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.925974 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\") pod \"e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:29.963536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:29.963511 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:30.409659 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:30.409631 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:49:30.412027 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:49:30.411998 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9909f7ed_0a52_4248_ac9f_94889dcfd72e.slice/crio-ee802b1107ee4edb94b0896750f516fcd92282e117c044d4cf4143c5384c4349 WatchSource:0}: Error finding container ee802b1107ee4edb94b0896750f516fcd92282e117c044d4cf4143c5384c4349: Status 404 returned error can't find the container with id ee802b1107ee4edb94b0896750f516fcd92282e117c044d4cf4143c5384c4349 Oct 30 19:49:30.414790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:30.414759 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:30.414891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:30.414820 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:30.414891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:30.414854 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:31.308454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:31.308418 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerStarted","Data":"3a62f146deeda15e07fc33102798d3ba639bf393ac751db2bfff71ef3d0cd83a"} Oct 30 19:49:31.308867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:31.308469 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerStarted","Data":"ee802b1107ee4edb94b0896750f516fcd92282e117c044d4cf4143c5384c4349"} Oct 30 19:49:31.478542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:31.478482 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podStartSLOduration=2.478453374 podStartE2EDuration="2.478453374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:31.47410614 +0000 UTC m=+3807.976016625" watchObservedRunningTime="2026-10-30 19:49:31.478453374 +0000 UTC m=+3807.980363862" Oct 30 19:49:31.964722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:31.964680 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:31.970794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:31.970769 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:32.316924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:32.316850 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:32.317800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:32.317776 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:49:35.431263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.431230 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f"] Oct 30 19:49:35.435783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.435761 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.486625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.486599 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:35.532853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.532804 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:35.532853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.532858 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:35.533080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.532887 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:49:35.535500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.535470 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.535597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.535534 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.535664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.535640 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zshp\" (UniqueName: \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.536002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.535985 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f"] Oct 30 19:49:35.636369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.636331 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.636540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.636375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zshp\" (UniqueName: \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.636609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.636561 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.637228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.637185 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.638687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.638668 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.693616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.693590 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zshp\" (UniqueName: \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\") pod \"maas-api-revoke-keys-e2e-autores-f858d4-k5g5f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:35.750320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:35.750297 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:36.129188 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:36.129160 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f"] Oct 30 19:49:36.131144 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:49:36.131120 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e7a813_c656_4cce_ae2e_0e2f5edb8b3f.slice/crio-18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a WatchSource:0}: Error finding container 18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a: Status 404 returned error can't find the container with id 18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a Oct 30 19:49:36.343368 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:36.343337 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" event={"ID":"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f","Type":"ContainerStarted","Data":"18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a"} Oct 30 19:49:37.352907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:37.352828 4789 generic.go:356] "Generic (PLEG): container finished" podID="51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" containerID="4e7f889c9a965615020cc499f0969661b1397f8259d7cae2945251d60ee2b34d" exitCode=0 Oct 30 19:49:37.352907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:37.352871 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" event={"ID":"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f","Type":"ContainerDied","Data":"4e7f889c9a965615020cc499f0969661b1397f8259d7cae2945251d60ee2b34d"} Oct 30 19:49:38.382691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.382656 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:49:38.386804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.386782 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.435911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.435887 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-660b59-dockercfg-vks52\"" Oct 30 19:49:38.462339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.462293 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8gg\" (UniqueName: \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.462543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.462473 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.511200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.511160 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:49:38.511650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.511564 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-660b59\"" Oct 30 19:49:38.548256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.548231 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:38.563778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.563531 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\") pod \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " Oct 30 19:49:38.563778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.563689 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\" (UniqueName: \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\") pod \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " Oct 30 19:49:38.563778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.563771 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\") pod \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\" (UID: \"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f\") " Oct 30 19:49:38.564199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.564168 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.564380 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.564332 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8gg\" (UniqueName: \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.564623 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.564544 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca" pod "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" (UID: "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:38.565132 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.565098 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.566320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.566295 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp" pod "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" (UID: "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f"). InnerVolumeSpecName "kube-api-access-2zshp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:38.566428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.566411 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca" pod "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" (UID: "51e7a813-c656-4cce-ae2e-0e2f5edb8b3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:38.567947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.567922 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:49:38.569491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.569472 4789 cpu_manager.go:414] "containerMap: removing container" podUID="51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" containerName="revoke-keys" Oct 30 19:49:38.569491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.569491 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" containerName="revoke-keys" Oct 30 19:49:38.569633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.569623 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" containerName="revoke-keys" Oct 30 19:49:38.573181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.573151 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.660312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.660245 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:49:38.666270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.666238 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.666416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.666324 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbkz\" (UniqueName: \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.666416 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.666406 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:38.666558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.666423 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zshp\" (UniqueName: \"kubernetes.io/projected/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-kube-api-access-2zshp\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:38.666558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.666437 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:38.678589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.678565 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8gg\" (UniqueName: \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\") pod \"payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.702822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.702790 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:38.767858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.767823 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.767974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.767919 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbkz\" (UniqueName: \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.768934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.768915 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.826990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.826960 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbkz\" (UniqueName: \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\") pod \"payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:38.901658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:38.901632 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:39.082430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.082406 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:49:39.084136 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:49:39.084111 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249b213b_d43c_475a_9afa_c5cef977a329.slice/crio-93394b35313125f7f3561620825fd1f15f183c622be58ffd308bd589dd2290bd WatchSource:0}: Error finding container 93394b35313125f7f3561620825fd1f15f183c622be58ffd308bd589dd2290bd: Status 404 returned error can't find the container with id 93394b35313125f7f3561620825fd1f15f183c622be58ffd308bd589dd2290bd Oct 30 19:49:39.278250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.278227 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:49:39.279349 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:49:39.279325 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e238f0b_aac5_4e07_a5c2_46cabac5eb4a.slice/crio-60eb4904e371f0de9ddc1ce39af9a5be1148fb12a5eea8107395dd5cc8fe65e0 WatchSource:0}: Error finding container 60eb4904e371f0de9ddc1ce39af9a5be1148fb12a5eea8107395dd5cc8fe65e0: Status 404 returned error can't find the container with id 60eb4904e371f0de9ddc1ce39af9a5be1148fb12a5eea8107395dd5cc8fe65e0 Oct 30 19:49:39.368435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.368405 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" event={"ID":"249b213b-d43c-475a-9afa-c5cef977a329","Type":"ContainerStarted","Data":"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8"} Oct 30 19:49:39.368559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.368461 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" event={"ID":"249b213b-d43c-475a-9afa-c5cef977a329","Type":"ContainerStarted","Data":"93394b35313125f7f3561620825fd1f15f183c622be58ffd308bd589dd2290bd"} Oct 30 19:49:39.368630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.368618 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:39.369738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.369721 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" Oct 30 19:49:39.369844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.369754 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f" event={"ID":"51e7a813-c656-4cce-ae2e-0e2f5edb8b3f","Type":"ContainerDied","Data":"18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a"} Oct 30 19:49:39.369844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.369799 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ad70e193fbd327c9249ccfab699d50b1a9ae9003fc97bc6d0f39ff23c0854a" Oct 30 19:49:39.371574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.371553 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" event={"ID":"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a","Type":"ContainerStarted","Data":"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0"} Oct 30 19:49:39.371888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.371590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" event={"ID":"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a","Type":"ContainerStarted","Data":"60eb4904e371f0de9ddc1ce39af9a5be1148fb12a5eea8107395dd5cc8fe65e0"} Oct 30 19:49:39.371888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.371636 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:39.479745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.479693 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" podStartSLOduration=1.4796807570000001 podStartE2EDuration="1.479680757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:39.478748138 +0000 UTC m=+3815.980658624" watchObservedRunningTime="2026-10-30 19:49:39.479680757 +0000 UTC m=+3815.981591243" Oct 30 19:49:39.640550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:39.640443 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" podStartSLOduration=1.6404281649999999 podStartE2EDuration="1.640428165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:39.637825836 +0000 UTC m=+3816.139736322" watchObservedRunningTime="2026-10-30 19:49:39.640428165 +0000 UTC m=+3816.142338650" Oct 30 19:49:42.590566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.590488 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerName="main" containerID="cri-o://8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d" gracePeriod=45 Oct 30 19:49:42.882728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.882703 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:49:42.915260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915222 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.915386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915321 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.915465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915389 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\" (UniqueName: \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.915605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915581 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.915605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915588 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:42.915759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915651 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.915759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915705 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\" (UniqueName: \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\") pod \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\" (UID: \"181096a4-aff4-406f-ac5d-f6b6c3d09caf\") " Oct 30 19:49:42.916229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915907 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:42.916335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.915979 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:42.916335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.916188 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:42.917871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.917617 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:42.917989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.917937 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "kube-api-access-8v2d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:42.918168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:42.918140 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs" pod "181096a4-aff4-406f-ac5d-f6b6c3d09caf" (UID: "181096a4-aff4-406f-ac5d-f6b6c3d09caf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:43.017957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.017928 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.018092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.017962 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v2d6\" (UniqueName: \"kubernetes.io/projected/181096a4-aff4-406f-ac5d-f6b6c3d09caf-kube-api-access-8v2d6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.018092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.017977 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.018092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.017992 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/181096a4-aff4-406f-ac5d-f6b6c3d09caf-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.018092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.018008 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/181096a4-aff4-406f-ac5d-f6b6c3d09caf-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:43.398990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.398960 4789 generic.go:356] "Generic (PLEG): container finished" podID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerID="8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d" exitCode=0 Oct 30 19:49:43.399193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.398997 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" Oct 30 19:49:43.399193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.399036 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" event={"ID":"181096a4-aff4-406f-ac5d-f6b6c3d09caf","Type":"ContainerDied","Data":"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d"} Oct 30 19:49:43.399193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.399070 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" event={"ID":"181096a4-aff4-406f-ac5d-f6b6c3d09caf","Type":"ContainerDied","Data":"aa832584de10da723a762a7f411da670e38aeaa6f3a5364c817ccdd477738b4b"} Oct 30 19:49:43.399193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.399089 4789 scope.go:122] "RemoveContainer" containerID="8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d" Oct 30 19:49:43.411167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.411142 4789 scope.go:122] "RemoveContainer" containerID="8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d" Oct 30 19:49:43.411473 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:43.411441 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d\": container with ID starting with 8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d not found: ID does not exist" containerID="8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d" Oct 30 19:49:43.411562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.411505 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d"} err="failed to get container status \"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d\": rpc error: code = NotFound desc = could not find container \"8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d\": container with ID starting with 8f7a39f0c10ec28d71575ff60b157c750279fcd5f4328a9372869401f7df9f4d not found: ID does not exist" Oct 30 19:49:43.555873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.555848 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:49:43.567028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.567002 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht"] Oct 30 19:49:43.668985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.668926 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" path="/var/lib/kubelet/pods/181096a4-aff4-406f-ac5d-f6b6c3d09caf/volumes" Oct 30 19:49:43.854898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:43.854865 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-c01685/auth-test-model-c01685-kserve-7dffbdd585-7kxht" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.247:8000/ready\": context deadline exceeded" Oct 30 19:49:48.197589 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:48.197556 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:49:50.381825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:50.381798 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:49:50.382236 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:50.381863 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:49:54.782139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.782095 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7393-d62905"] Oct 30 19:49:54.782739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.782719 4789 cpu_manager.go:414] "containerMap: removing container" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerName="main" Oct 30 19:49:54.782739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.782737 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerName="main" Oct 30 19:49:54.782914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.782851 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="181096a4-aff4-406f-ac5d-f6b6c3d09caf" containerName="main" Oct 30 19:49:54.787692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.787673 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:54.788918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.788893 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dq96\" (UniqueName: \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\") pod \"test-curl-7393-d62905\" (UID: \"813edcea-516f-4507-a7ed-90f668601492\") " pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:54.837670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.837646 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7hnn9\"" Oct 30 19:49:54.838460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.838438 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7393-d62905"] Oct 30 19:49:54.889795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.889765 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dq96\" (UniqueName: \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\") pod \"test-curl-7393-d62905\" (UID: \"813edcea-516f-4507-a7ed-90f668601492\") " pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:54.947747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:54.947717 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dq96\" (UniqueName: \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\") pod \"test-curl-7393-d62905\" (UID: \"813edcea-516f-4507-a7ed-90f668601492\") " pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:55.127532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:55.127434 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:55.504426 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:55.504398 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7393-d62905"] Oct 30 19:49:55.513240 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:49:55.507364 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813edcea_516f_4507_a7ed_90f668601492.slice/crio-ae1bf40c899704233caa61148f5fb91c303b364460c154adb6b74f0bb9ed50d5 WatchSource:0}: Error finding container ae1bf40c899704233caa61148f5fb91c303b364460c154adb6b74f0bb9ed50d5: Status 404 returned error can't find the container with id ae1bf40c899704233caa61148f5fb91c303b364460c154adb6b74f0bb9ed50d5 Oct 30 19:49:56.499225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:56.499067 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7393-d62905" event={"ID":"813edcea-516f-4507-a7ed-90f668601492","Type":"ContainerStarted","Data":"798a7a97349d5f98f3b2d255f169538d5ec50b9b80be5481a03f53e002954cf6"} Oct 30 19:49:56.499225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:56.499113 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7393-d62905" event={"ID":"813edcea-516f-4507-a7ed-90f668601492","Type":"ContainerStarted","Data":"ae1bf40c899704233caa61148f5fb91c303b364460c154adb6b74f0bb9ed50d5"} Oct 30 19:49:56.607786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:56.607717 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7393-d62905" podStartSLOduration=2.607699297 podStartE2EDuration="2.607699297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:56.606437798 +0000 UTC m=+3833.108348285" watchObservedRunningTime="2026-10-30 19:49:56.607699297 +0000 UTC m=+3833.109609788" Oct 30 19:49:56.677366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:56.677327 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:49:56.677518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:56.677442 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:49:58.050540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.050499 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7393-d62905"] Oct 30 19:49:58.055552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.055527 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7393-d62905"] Oct 30 19:49:58.354189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.354165 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p"] Oct 30 19:49:58.355074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.355052 4789 cpu_manager.go:414] "containerMap: removing container" podUID="813edcea-516f-4507-a7ed-90f668601492" containerName="test-curl-7393-d62905" Oct 30 19:49:58.355122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.355079 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813edcea-516f-4507-a7ed-90f668601492" containerName="test-curl-7393-d62905" Oct 30 19:49:58.355262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.355249 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="813edcea-516f-4507-a7ed-90f668601492" containerName="test-curl-7393-d62905" Oct 30 19:49:58.358365 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.358347 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.409621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.409594 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:58.433969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.433945 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.434124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.434101 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.434217 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.434189 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5kr\" (UniqueName: \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.455863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.455835 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:49:58.456099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.456083 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:58.456199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.456109 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:49:58.459189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.459165 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p"] Oct 30 19:49:58.518552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.518495 4789 generic.go:356] "Generic (PLEG): container finished" podID="98209702-5b61-426c-9470-168fa30d8f04" containerID="6c160a6f7002f2d9c9393efae0ddef1a2000bc5ed306f7535b054d84056a09ab" exitCode=0 Oct 30 19:49:58.518649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.518578 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerDied","Data":"6c160a6f7002f2d9c9393efae0ddef1a2000bc5ed306f7535b054d84056a09ab"} Oct 30 19:49:58.518747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.518731 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7393-d62905" podUID="813edcea-516f-4507-a7ed-90f668601492" containerName="test-curl-7393-d62905" containerID="cri-o://798a7a97349d5f98f3b2d255f169538d5ec50b9b80be5481a03f53e002954cf6" gracePeriod=2 Oct 30 19:49:58.519382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.518999 4789 scope.go:122] "RemoveContainer" containerID="6c160a6f7002f2d9c9393efae0ddef1a2000bc5ed306f7535b054d84056a09ab" Oct 30 19:49:58.520866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.520836 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:58.520969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.520900 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:58.520969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.520930 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:58.535575 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.535555 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.535746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.535728 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.535845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.535819 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5kr\" (UniqueName: \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.536748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.536721 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.537991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.537967 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.593691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.593671 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5kr\" (UniqueName: \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\") pod \"maas-api-revoke-sub-e9c8d2a6-nkg6p\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.682004 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:58.681966 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7393-d62905\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="813edcea-516f-4507-a7ed-90f668601492" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:58.726559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:58.726525 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:49:58.730182 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:58.730152 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7393-d62905\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="813edcea-516f-4507-a7ed-90f668601492" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:49:59.122352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.122310 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p"] Oct 30 19:49:59.556041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.553387 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerStarted","Data":"396fdf5f274bf46e8cb370ee4567d8a898db755e108d65f1a787842faef2e47e"} Oct 30 19:49:59.557853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.556611 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" event={"ID":"2725ae76-8380-4677-aed3-7a6486c46319","Type":"ContainerStarted","Data":"72012e017e90b6bb0b2c4d8e0ba31e66f8941ff0e078d3fb391a7625fe4ba881"} Oct 30 19:49:59.557853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.556965 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:49:59.561701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.561679 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:49:59.561888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.561864 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:49:59.561969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:49:59.561919 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:49:59.632265 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:49:59.615153 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7393-d62905\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="813edcea-516f-4507-a7ed-90f668601492" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:50:00.560843 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.560811 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:50:00.561195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.560871 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:50:00.568547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.568521 4789 generic.go:356] "Generic (PLEG): container finished" podID="813edcea-516f-4507-a7ed-90f668601492" containerID="798a7a97349d5f98f3b2d255f169538d5ec50b9b80be5481a03f53e002954cf6" exitCode=137 Oct 30 19:50:00.570306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.570283 4789 generic.go:356] "Generic (PLEG): container finished" podID="2725ae76-8380-4677-aed3-7a6486c46319" containerID="45780354ed22032c76f8024f84599f8d154f28ed97ed791d23e41db2a4069b64" exitCode=0 Oct 30 19:50:00.570402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.570369 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" event={"ID":"2725ae76-8380-4677-aed3-7a6486c46319","Type":"ContainerDied","Data":"45780354ed22032c76f8024f84599f8d154f28ed97ed791d23e41db2a4069b64"} Oct 30 19:50:00.880289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.880265 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:50:00.970113 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.970082 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\" (UniqueName: \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\") pod \"813edcea-516f-4507-a7ed-90f668601492\" (UID: \"813edcea-516f-4507-a7ed-90f668601492\") " Oct 30 19:50:00.972127 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:00.972105 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96" pod "813edcea-516f-4507-a7ed-90f668601492" (UID: "813edcea-516f-4507-a7ed-90f668601492"). InnerVolumeSpecName "kube-api-access-4dq96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:01.071772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.071733 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dq96\" (UniqueName: \"kubernetes.io/projected/813edcea-516f-4507-a7ed-90f668601492-kube-api-access-4dq96\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.560784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.560743 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:50:01.560996 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.560793 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:50:01.578009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.577986 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:50:01.578126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.578039 4789 scope.go:122] "RemoveContainer" containerID="798a7a97349d5f98f3b2d255f169538d5ec50b9b80be5481a03f53e002954cf6" Oct 30 19:50:01.636126 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:50:01.636083 4789 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7393-d62905\" is forbidden: User \"system:node:ip-10-0-136-46.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-46.us-west-2.compute.internal' and this object" podUID="813edcea-516f-4507-a7ed-90f668601492" pod="opendatahub/test-curl-7393-d62905" Oct 30 19:50:01.674115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.674078 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813edcea-516f-4507-a7ed-90f668601492" path="/var/lib/kubelet/pods/813edcea-516f-4507-a7ed-90f668601492/volumes" Oct 30 19:50:01.756478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.756454 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:50:01.780268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.780240 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\") pod \"2725ae76-8380-4677-aed3-7a6486c46319\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " Oct 30 19:50:01.780449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.780430 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\" (UniqueName: \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\") pod \"2725ae76-8380-4677-aed3-7a6486c46319\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " Oct 30 19:50:01.780523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.780506 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\") pod \"2725ae76-8380-4677-aed3-7a6486c46319\" (UID: \"2725ae76-8380-4677-aed3-7a6486c46319\") " Oct 30 19:50:01.780610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.780593 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca" pod "2725ae76-8380-4677-aed3-7a6486c46319" (UID: "2725ae76-8380-4677-aed3-7a6486c46319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:01.781013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.780989 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.782572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.782541 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca" pod "2725ae76-8380-4677-aed3-7a6486c46319" (UID: "2725ae76-8380-4677-aed3-7a6486c46319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:01.782572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.782565 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr" pod "2725ae76-8380-4677-aed3-7a6486c46319" (UID: "2725ae76-8380-4677-aed3-7a6486c46319"). InnerVolumeSpecName "kube-api-access-9g5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:01.882193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.882130 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5kr\" (UniqueName: \"kubernetes.io/projected/2725ae76-8380-4677-aed3-7a6486c46319-kube-api-access-9g5kr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.882193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.882158 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2725ae76-8380-4677-aed3-7a6486c46319-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:01.927297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.927261 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c"] Oct 30 19:50:01.928139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.928119 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2725ae76-8380-4677-aed3-7a6486c46319" containerName="revoke-keys" Oct 30 19:50:01.928202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.928142 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2725ae76-8380-4677-aed3-7a6486c46319" containerName="revoke-keys" Oct 30 19:50:01.928391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.928357 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2725ae76-8380-4677-aed3-7a6486c46319" containerName="revoke-keys" Oct 30 19:50:01.931355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.931335 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:01.982883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.982856 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:01.983000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.982919 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:01.983138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.983112 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qvvw\" (UniqueName: \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:01.988954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:01.988931 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c"] Oct 30 19:50:02.084592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.084564 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.084791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.084655 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qvvw\" (UniqueName: \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.084791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.084742 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.085388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.085342 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.086863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.086841 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.142847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.142787 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qvvw\" (UniqueName: \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\") pod \"maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.244601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.244567 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:02.560885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.560846 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:50:02.561083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.560901 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:50:02.586747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.586723 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" Oct 30 19:50:02.586874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.586764 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p" event={"ID":"2725ae76-8380-4677-aed3-7a6486c46319","Type":"ContainerDied","Data":"72012e017e90b6bb0b2c4d8e0ba31e66f8941ff0e078d3fb391a7625fe4ba881"} Oct 30 19:50:02.586874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.586805 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72012e017e90b6bb0b2c4d8e0ba31e66f8941ff0e078d3fb391a7625fe4ba881" Oct 30 19:50:02.640374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:02.640351 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c"] Oct 30 19:50:02.641841 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:50:02.641809 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd4e415_a67c_46b8_8772_2d0c4ef258ac.slice/crio-74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b WatchSource:0}: Error finding container 74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b: Status 404 returned error can't find the container with id 74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b Oct 30 19:50:03.561400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:03.561356 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:50:03.561789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:03.561416 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:50:03.618892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:03.618855 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" event={"ID":"4dd4e415-a67c-46b8-8772-2d0c4ef258ac","Type":"ContainerStarted","Data":"fea694f46c8323c6f8431126bb61e71d4e8750e668848992515420c3e13790a0"} Oct 30 19:50:03.619092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:03.618920 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" event={"ID":"4dd4e415-a67c-46b8-8772-2d0c4ef258ac","Type":"ContainerStarted","Data":"74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b"} Oct 30 19:50:03.731229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:03.731158 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" podStartSLOduration=2.731144623 podStartE2EDuration="2.731144623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:03.729039435 +0000 UTC m=+3840.230949921" watchObservedRunningTime="2026-10-30 19:50:03.731144623 +0000 UTC m=+3840.233055109" Oct 30 19:50:04.561476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:04.561439 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 19:50:04.561942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:04.561492 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 19:50:04.629910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:04.629877 4789 generic.go:356] "Generic (PLEG): container finished" podID="4dd4e415-a67c-46b8-8772-2d0c4ef258ac" containerID="fea694f46c8323c6f8431126bb61e71d4e8750e668848992515420c3e13790a0" exitCode=0 Oct 30 19:50:04.630068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:04.629914 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" event={"ID":"4dd4e415-a67c-46b8-8772-2d0c4ef258ac","Type":"ContainerDied","Data":"fea694f46c8323c6f8431126bb61e71d4e8750e668848992515420c3e13790a0"} Oct 30 19:50:05.565569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.565541 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:50:05.566059 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.566041 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:50:05.566800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.566783 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:50:05.786252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.786225 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:05.853582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.853517 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\") pod \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " Oct 30 19:50:05.853582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.853555 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\" (UniqueName: \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\") pod \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " Oct 30 19:50:05.853830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.853613 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\") pod \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\" (UID: \"4dd4e415-a67c-46b8-8772-2d0c4ef258ac\") " Oct 30 19:50:05.853882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.853866 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca" pod "4dd4e415-a67c-46b8-8772-2d0c4ef258ac" (UID: "4dd4e415-a67c-46b8-8772-2d0c4ef258ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:05.854117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.854097 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:05.855723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.855685 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca" pod "4dd4e415-a67c-46b8-8772-2d0c4ef258ac" (UID: "4dd4e415-a67c-46b8-8772-2d0c4ef258ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:05.855825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.855761 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw" pod "4dd4e415-a67c-46b8-8772-2d0c4ef258ac" (UID: "4dd4e415-a67c-46b8-8772-2d0c4ef258ac"). InnerVolumeSpecName "kube-api-access-2qvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:05.955638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.955612 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qvvw\" (UniqueName: \"kubernetes.io/projected/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-kube-api-access-2qvvw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:05.955638 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:05.955637 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd4e415-a67c-46b8-8772-2d0c4ef258ac-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:06.644073 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:06.644040 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" Oct 30 19:50:06.644563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:06.644071 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c" event={"ID":"4dd4e415-a67c-46b8-8772-2d0c4ef258ac","Type":"ContainerDied","Data":"74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b"} Oct 30 19:50:06.644563 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:06.644121 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c5a9929df2c5df0eefc24b5276a14c140e4a00640604f0eb00b457a194084b" Oct 30 19:50:09.068472 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.068435 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd"] Oct 30 19:50:09.069800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.069776 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4dd4e415-a67c-46b8-8772-2d0c4ef258ac" containerName="revoke-keys" Oct 30 19:50:09.069947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.069936 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd4e415-a67c-46b8-8772-2d0c4ef258ac" containerName="revoke-keys" Oct 30 19:50:09.070191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.070180 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd4e415-a67c-46b8-8772-2d0c4ef258ac" containerName="revoke-keys" Oct 30 19:50:09.074163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.074134 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.171250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.171223 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:09.171465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.171439 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:50:09.171548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.171477 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:50:09.171548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.171477 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:50:09.174491 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.174471 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd"] Oct 30 19:50:09.188831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.188805 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.188941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.188922 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.189011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.188992 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glx5\" (UniqueName: \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.289481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.289455 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glx5\" (UniqueName: \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.289610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.289534 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.289610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.289601 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.290223 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.290195 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.291874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.291856 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.348936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.348886 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glx5\" (UniqueName: \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\") pod \"maas-api-revoke-sub-9749efc7-9vqsd\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.394585 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.394558 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:09.771062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:09.771013 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd"] Oct 30 19:50:09.772152 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:50:09.772116 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba6458b_ead0_4ac4_b6ba_bc1559b10b43.slice/crio-d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e WatchSource:0}: Error finding container d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e: Status 404 returned error can't find the container with id d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e Oct 30 19:50:10.683018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:10.682991 4789 generic.go:356] "Generic (PLEG): container finished" podID="5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" containerID="a837d1a8a6008cb664bca95eb904f95047f9b218d365c6151a37438b042c97da" exitCode=0 Oct 30 19:50:10.683386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:10.683077 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" event={"ID":"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43","Type":"ContainerDied","Data":"a837d1a8a6008cb664bca95eb904f95047f9b218d365c6151a37438b042c97da"} Oct 30 19:50:10.683386 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:10.683120 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" event={"ID":"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43","Type":"ContainerStarted","Data":"d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e"} Oct 30 19:50:11.854068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:11.854041 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:12.025611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.025526 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\") pod \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " Oct 30 19:50:12.025611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.025575 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\") pod \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " Oct 30 19:50:12.025841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.025727 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\" (UniqueName: \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\") pod \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\" (UID: \"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43\") " Oct 30 19:50:12.026011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.025989 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca" pod "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" (UID: "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:12.026191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.026169 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:12.028103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.028074 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca" pod "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" (UID: "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:12.028286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.028197 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5" pod "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" (UID: "5ba6458b-ead0-4ac4-b6ba-bc1559b10b43"). InnerVolumeSpecName "kube-api-access-7glx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:12.126701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.126672 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7glx5\" (UniqueName: \"kubernetes.io/projected/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-kube-api-access-7glx5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:12.126701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.126697 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:12.697327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.697292 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" Oct 30 19:50:12.697327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.697305 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd" event={"ID":"5ba6458b-ead0-4ac4-b6ba-bc1559b10b43","Type":"ContainerDied","Data":"d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e"} Oct 30 19:50:12.697527 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:12.697348 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30bcf66bd316bb94e9db6b32349cb935d97ff7295a19ec81381a347b16b4d9e" Oct 30 19:50:13.151816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:13.151738 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f"] Oct 30 19:50:13.154757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:13.154735 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f858d4-k5g5f"] Oct 30 19:50:13.322197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:13.322169 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:50:13.433707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:13.433684 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:50:13.669519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:13.669485 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e7a813-c656-4cce-ae2e-0e2f5edb8b3f" path="/var/lib/kubelet/pods/51e7a813-c656-4cce-ae2e-0e2f5edb8b3f/volumes" Oct 30 19:50:14.579727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:14.579689 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" podUID="ff47f656-2879-4cd2-827d-53aa86765bce" containerName="main" containerID="cri-o://238840831fef6ca42b1fedf1931d926cb1d3ab2f0368aedf3b8ed22fe428d7a1" gracePeriod=45 Oct 30 19:50:14.727568 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:14.727520 4789 generic.go:356] "Generic (PLEG): container finished" podID="ff47f656-2879-4cd2-827d-53aa86765bce" containerID="238840831fef6ca42b1fedf1931d926cb1d3ab2f0368aedf3b8ed22fe428d7a1" exitCode=0 Oct 30 19:50:14.727705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:14.727561 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" event={"ID":"ff47f656-2879-4cd2-827d-53aa86765bce","Type":"ContainerDied","Data":"238840831fef6ca42b1fedf1931d926cb1d3ab2f0368aedf3b8ed22fe428d7a1"} Oct 30 19:50:14.895233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:14.895179 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:50:15.060286 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060258 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\" (UniqueName: \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060351 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060525 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060566 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:15.060601 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060592 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060723 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060671 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060764 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:15.060860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060785 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\") pod \"ff47f656-2879-4cd2-827d-53aa86765bce\" (UID: \"ff47f656-2879-4cd2-827d-53aa86765bce\") " Oct 30 19:50:15.060965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.060946 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:15.061308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.061276 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.061308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.061298 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.061308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.061310 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.062755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.062731 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "kube-api-access-8mf5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:15.063081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.063063 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:50:15.063146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.063077 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs" pod "ff47f656-2879-4cd2-827d-53aa86765bce" (UID: "ff47f656-2879-4cd2-827d-53aa86765bce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:15.162853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.162788 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff47f656-2879-4cd2-827d-53aa86765bce-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.162853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.162820 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mf5m\" (UniqueName: \"kubernetes.io/projected/ff47f656-2879-4cd2-827d-53aa86765bce-kube-api-access-8mf5m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.162853 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.162831 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff47f656-2879-4cd2-827d-53aa86765bce-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:15.734564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.734537 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" Oct 30 19:50:15.735123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.734567 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727" event={"ID":"ff47f656-2879-4cd2-827d-53aa86765bce","Type":"ContainerDied","Data":"2584389081daf6cc634347143f4b08f16c47801f4911d7ec519620fc7b65adba"} Oct 30 19:50:15.735123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.734613 4789 scope.go:122] "RemoveContainer" containerID="238840831fef6ca42b1fedf1931d926cb1d3ab2f0368aedf3b8ed22fe428d7a1" Oct 30 19:50:15.897028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.897002 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:50:15.942850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:15.942799 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f858d4/autores-model-f858d4-kserve-6c9549b697-jr727"] Oct 30 19:50:17.668761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:17.668724 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff47f656-2879-4cd2-827d-53aa86765bce" path="/var/lib/kubelet/pods/ff47f656-2879-4cd2-827d-53aa86765bce/volumes" Oct 30 19:50:18.196833 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:50:18.196799 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:50:18.323393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.323352 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" podUID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" containerName="payload-processing" containerID="cri-o://79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023" gracePeriod=25 Oct 30 19:50:18.434425 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.434387 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" podUID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" containerName="payload-pre-processing" containerID="cri-o://a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69" gracePeriod=25 Oct 30 19:50:18.613530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.613501 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:50:18.702429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.702373 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\") pod \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " Oct 30 19:50:18.702836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.702467 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\" (UniqueName: \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\") pod \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\" (UID: \"ee2c5ae7-7ea6-4527-943a-34bc7d34efea\") " Oct 30 19:50:18.702836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.702699 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume" pod "ee2c5ae7-7ea6-4527-943a-34bc7d34efea" (UID: "ee2c5ae7-7ea6-4527-943a-34bc7d34efea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:18.702836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.702799 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:18.703336 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.703316 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:50:18.704528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.704501 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd" pod "ee2c5ae7-7ea6-4527-943a-34bc7d34efea" (UID: "ee2c5ae7-7ea6-4527-943a-34bc7d34efea"). InnerVolumeSpecName "kube-api-access-8m5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:18.755987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.755955 4789 generic.go:356] "Generic (PLEG): container finished" podID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" containerID="a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69" exitCode=0 Oct 30 19:50:18.756116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.756004 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" Oct 30 19:50:18.756116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.756040 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" event={"ID":"77819d6d-5ced-4dac-b0c6-2fc83d4fd642","Type":"ContainerDied","Data":"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69"} Oct 30 19:50:18.756116 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.756106 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6" event={"ID":"77819d6d-5ced-4dac-b0c6-2fc83d4fd642","Type":"ContainerDied","Data":"33e3711231ca06ddafc23757d01cb8cd2b32c40a9a2201081ef4d932c751d4f2"} Oct 30 19:50:18.756263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.756139 4789 scope.go:122] "RemoveContainer" containerID="a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69" Oct 30 19:50:18.757881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.757860 4789 generic.go:356] "Generic (PLEG): container finished" podID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" containerID="79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023" exitCode=0 Oct 30 19:50:18.757983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.757888 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" event={"ID":"ee2c5ae7-7ea6-4527-943a-34bc7d34efea","Type":"ContainerDied","Data":"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023"} Oct 30 19:50:18.757983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.757911 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" Oct 30 19:50:18.757983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.757926 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8" event={"ID":"ee2c5ae7-7ea6-4527-943a-34bc7d34efea","Type":"ContainerDied","Data":"66ef110aa9d305159146dddeb0e8e4b9b7764f80a974735abc5efb9507cb98a1"} Oct 30 19:50:18.766410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.766391 4789 scope.go:122] "RemoveContainer" containerID="a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69" Oct 30 19:50:18.766686 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:50:18.766667 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69\": container with ID starting with a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69 not found: ID does not exist" containerID="a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69" Oct 30 19:50:18.766748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.766692 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69"} err="failed to get container status \"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69\": rpc error: code = NotFound desc = could not find container \"a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69\": container with ID starting with a12f2fd722b9872584afe399b39cecffd4ba7a6b4e93fa02d574fecdb439be69 not found: ID does not exist" Oct 30 19:50:18.766748 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.766710 4789 scope.go:122] "RemoveContainer" containerID="79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023" Oct 30 19:50:18.775931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.775915 4789 scope.go:122] "RemoveContainer" containerID="79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023" Oct 30 19:50:18.776168 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:50:18.776148 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023\": container with ID starting with 79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023 not found: ID does not exist" containerID="79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023" Oct 30 19:50:18.776252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.776177 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023"} err="failed to get container status \"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023\": rpc error: code = NotFound desc = could not find container \"79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023\": container with ID starting with 79e6c7b13f71e89ee8da33bc35f795f8308ab3fda9f87960a6d30a790461d023 not found: ID does not exist" Oct 30 19:50:18.803760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.803736 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\") pod \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " Oct 30 19:50:18.803968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.803949 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\" (UniqueName: \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\") pod \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\" (UID: \"77819d6d-5ced-4dac-b0c6-2fc83d4fd642\") " Oct 30 19:50:18.804162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.804142 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume" pod "77819d6d-5ced-4dac-b0c6-2fc83d4fd642" (UID: "77819d6d-5ced-4dac-b0c6-2fc83d4fd642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:18.804334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.804321 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5bd\" (UniqueName: \"kubernetes.io/projected/ee2c5ae7-7ea6-4527-943a-34bc7d34efea-kube-api-access-8m5bd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:18.804414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.804335 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:18.806044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.806021 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk" pod "77819d6d-5ced-4dac-b0c6-2fc83d4fd642" (UID: "77819d6d-5ced-4dac-b0c6-2fc83d4fd642"). InnerVolumeSpecName "kube-api-access-x9cvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:18.905499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.905468 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9cvk\" (UniqueName: \"kubernetes.io/projected/77819d6d-5ced-4dac-b0c6-2fc83d4fd642-kube-api-access-x9cvk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:18.942672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.942639 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:50:18.946692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:18.946664 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a2f378-8498fd9c4d-rkmd8"] Oct 30 19:50:19.016518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.016433 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c"] Oct 30 19:50:19.021018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.020993 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a2f378-j547c"] Oct 30 19:50:19.170146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.170123 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:50:19.174952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.174928 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a2f378-68765c86b7-sw6j6"] Oct 30 19:50:19.668173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.668103 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd4e415-a67c-46b8-8772-2d0c4ef258ac" path="/var/lib/kubelet/pods/4dd4e415-a67c-46b8-8772-2d0c4ef258ac/volumes" Oct 30 19:50:19.668519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.668504 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" path="/var/lib/kubelet/pods/77819d6d-5ced-4dac-b0c6-2fc83d4fd642/volumes" Oct 30 19:50:19.668863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:19.668851 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" path="/var/lib/kubelet/pods/ee2c5ae7-7ea6-4527-943a-34bc7d34efea/volumes" Oct 30 19:50:22.055864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.055828 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:50:22.056749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056729 4789 cpu_manager.go:414] "containerMap: removing container" podUID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" containerName="payload-pre-processing" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056750 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" containerName="payload-pre-processing" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056760 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ff47f656-2879-4cd2-827d-53aa86765bce" containerName="main" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056765 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff47f656-2879-4cd2-827d-53aa86765bce" containerName="main" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056773 4789 cpu_manager.go:414] "containerMap: removing container" podUID="5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" containerName="revoke-keys" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056779 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" containerName="revoke-keys" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056797 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" containerName="payload-processing" Oct 30 19:50:22.056824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056802 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" containerName="payload-processing" Oct 30 19:50:22.057181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056907 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="77819d6d-5ced-4dac-b0c6-2fc83d4fd642" containerName="payload-pre-processing" Oct 30 19:50:22.057181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056915 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff47f656-2879-4cd2-827d-53aa86765bce" containerName="main" Oct 30 19:50:22.057181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056922 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" containerName="revoke-keys" Oct 30 19:50:22.057181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.056930 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2c5ae7-7ea6-4527-943a-34bc7d34efea" containerName="payload-processing" Oct 30 19:50:22.061837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.061818 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.111479 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.111445 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4d5083-openshift-default-dockercfg-pdh55\"" Oct 30 19:50:22.138868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.138838 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139166 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139268 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139329 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139359 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4hx\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139389 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139472 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139522 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.139579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.139547 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.162314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.162284 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:50:22.240746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240713 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.240918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240762 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.240918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240790 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.240918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240835 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240919 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.240970 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241029 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241056 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4hx\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241085 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241397 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241453 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241563 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.241891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.241634 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.242290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.242269 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.243885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.243858 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.244447 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.244422 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.308547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.308472 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4hx\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.335422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.330921 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\") pod \"e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.374724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.374694 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:22.826513 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.826481 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:50:22.827201 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:50:22.827169 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799befd8_3450_4228_8762_49775feb35a7.slice/crio-6ecbb02c0afaa1283303df38b34d086333802df3142b7fd9956501d5f2de363b WatchSource:0}: Error finding container 6ecbb02c0afaa1283303df38b34d086333802df3142b7fd9956501d5f2de363b: Status 404 returned error can't find the container with id 6ecbb02c0afaa1283303df38b34d086333802df3142b7fd9956501d5f2de363b Oct 30 19:50:22.830157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.830119 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:22.830300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.830203 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:22.830300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:22.830260 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:23.797241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:23.797173 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" event={"ID":"799befd8-3450-4228-8762-49775feb35a7","Type":"ContainerStarted","Data":"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f"} Oct 30 19:50:23.797693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:23.797267 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" event={"ID":"799befd8-3450-4228-8762-49775feb35a7","Type":"ContainerStarted","Data":"6ecbb02c0afaa1283303df38b34d086333802df3142b7fd9956501d5f2de363b"} Oct 30 19:50:23.960202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:23.960153 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" podStartSLOduration=1.960136323 podStartE2EDuration="1.960136323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:23.95777877 +0000 UTC m=+3860.459689254" watchObservedRunningTime="2026-10-30 19:50:23.960136323 +0000 UTC m=+3860.462046813" Oct 30 19:50:24.375813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:24.375781 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:24.381631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:24.381606 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:24.804290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:24.804260 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:24.805076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:24.805057 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:50:46.269840 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.269809 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:50:46.274204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.274187 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.325610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.325587 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4a36b7\"" Oct 30 19:50:46.330351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.330325 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:50:46.339818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.339795 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.339960 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.339915 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4j2\" (UniqueName: \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.371757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.371733 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4a36b7-dockercfg-cq9kc\"" Oct 30 19:50:46.441089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.441054 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.441276 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.441175 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4j2\" (UniqueName: \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.441961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.441937 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.499055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.499031 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4j2\" (UniqueName: \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\") pod \"payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.588310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.588241 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:46.962162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:46.962134 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:50:46.964551 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:50:46.964509 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b974bca_d824_4838_a08b_5f8304fc16c9.slice/crio-d8ba5efc5d90afe36f6d3822f519bb4a215a57d8fbbdcc131605566810ae297e WatchSource:0}: Error finding container d8ba5efc5d90afe36f6d3822f519bb4a215a57d8fbbdcc131605566810ae297e: Status 404 returned error can't find the container with id d8ba5efc5d90afe36f6d3822f519bb4a215a57d8fbbdcc131605566810ae297e Oct 30 19:50:47.955547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:47.955509 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" event={"ID":"3b974bca-d824-4838-a08b-5f8304fc16c9","Type":"ContainerStarted","Data":"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7"} Oct 30 19:50:47.955945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:47.955562 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" event={"ID":"3b974bca-d824-4838-a08b-5f8304fc16c9","Type":"ContainerStarted","Data":"d8ba5efc5d90afe36f6d3822f519bb4a215a57d8fbbdcc131605566810ae297e"} Oct 30 19:50:47.955945 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:47.955593 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:50:48.067126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:48.067076 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" podStartSLOduration=2.067062984 podStartE2EDuration="2.067062984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:48.064847515 +0000 UTC m=+3884.566758001" watchObservedRunningTime="2026-10-30 19:50:48.067062984 +0000 UTC m=+3884.568973513" Oct 30 19:50:48.200595 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:50:48.200549 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:50:58.966100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:50:58.966070 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:51:03.487450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.487419 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:03.492310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.492288 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.543780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.543743 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"kube-root-ca.crt\"" Oct 30 19:51:03.543780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.543756 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"openshift-service-ca.crt\"" Oct 30 19:51:03.543979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.543941 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"noten-model-4d5083-kserve-self-signed-certs\"" Oct 30 19:51:03.544035 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.544023 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"default-dockercfg-jgqsd\"" Oct 30 19:51:03.547616 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.547589 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:03.619965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.619925 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.620149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.619986 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpv2c\" (UniqueName: \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.620149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.620087 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.620318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.620179 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.620318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.620255 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.620318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.620301 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.720791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.720752 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.720985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.720825 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.720985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.720853 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.720985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.720903 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpv2c\" (UniqueName: \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.721174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.720989 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.721174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.721090 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.721174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.721145 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.721645 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.721618 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.721761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.721648 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.723369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.723341 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.769144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.769086 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"noten-model-4d5083-kserve-self-signed-certs\"" Oct 30 19:51:03.773540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.773517 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:03.824270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.824249 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"kube-root-ca.crt\"" Oct 30 19:51:03.875126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.875096 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"openshift-service-ca.crt\"" Oct 30 19:51:03.880074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:03.880036 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpv2c\" (UniqueName: \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\") pod \"noten-model-4d5083-kserve-855bd849cc-9dlr7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:04.153385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:04.153311 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4d5083\"/\"default-dockercfg-jgqsd\"" Oct 30 19:51:04.155422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:04.155407 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:04.525869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:04.525843 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:04.527869 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:51:04.527842 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48e6c818_b292_49a1_b6a2_c430ad1dc0b7.slice/crio-e9ed3922e5ac7f7c464090583624d43847d117e02a5f8dd3a33fd7dbb8712f28 WatchSource:0}: Error finding container e9ed3922e5ac7f7c464090583624d43847d117e02a5f8dd3a33fd7dbb8712f28: Status 404 returned error can't find the container with id e9ed3922e5ac7f7c464090583624d43847d117e02a5f8dd3a33fd7dbb8712f28 Oct 30 19:51:04.574910 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:04.574886 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:51:04.596145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:04.596122 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:51:05.074077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:05.074036 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" event={"ID":"48e6c818-b292-49a1-b6a2-c430ad1dc0b7","Type":"ContainerStarted","Data":"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb"} Oct 30 19:51:05.074296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:05.074092 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" event={"ID":"48e6c818-b292-49a1-b6a2-c430ad1dc0b7","Type":"ContainerStarted","Data":"e9ed3922e5ac7f7c464090583624d43847d117e02a5f8dd3a33fd7dbb8712f28"} Oct 30 19:51:05.155892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:05.155867 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:05.187028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:05.186983 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" podStartSLOduration=2.18696878 podStartE2EDuration="2.18696878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:05.18457543 +0000 UTC m=+3901.686485917" watchObservedRunningTime="2026-10-30 19:51:05.18696878 +0000 UTC m=+3901.688879265" Oct 30 19:51:14.156328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:14.156249 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:14.168989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:14.168966 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:15.159290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:15.159265 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:18.196950 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:51:18.196912 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:51:18.864950 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:18.864911 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:51:27.264780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.264753 4789 generic.go:356] "Generic (PLEG): container finished" podID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerID="3a62f146deeda15e07fc33102798d3ba639bf393ac751db2bfff71ef3d0cd83a" exitCode=0 Oct 30 19:51:27.265133 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.264817 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerDied","Data":"3a62f146deeda15e07fc33102798d3ba639bf393ac751db2bfff71ef3d0cd83a"} Oct 30 19:51:27.265193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.265178 4789 scope.go:122] "RemoveContainer" containerID="3a62f146deeda15e07fc33102798d3ba639bf393ac751db2bfff71ef3d0cd83a" Oct 30 19:51:27.266734 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.266706 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:27.266809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.266775 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:27.266846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.266809 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:27.518155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.518069 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:27.687836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.687801 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:51:27.688123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:27.688092 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" podUID="799befd8-3450-4228-8762-49775feb35a7" containerName="istio-proxy" containerID="cri-o://39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f" gracePeriod=30 Oct 30 19:51:28.273385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:28.273352 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerStarted","Data":"3cbc45f05e23446b52a837beff6c0d37e5d09805ad9c2cea305dfa33c0727265"} Oct 30 19:51:28.963720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:28.963688 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:51:28.965145 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:28.965115 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 19:51:28.965313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:28.965170 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 19:51:29.964644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:29.964611 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 19:51:29.965039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:29.964674 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 19:51:30.963976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:30.963940 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 19:51:30.964173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:30.963992 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 19:51:31.964664 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:31.964631 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 19:51:31.965167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:31.964690 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 19:51:32.320384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:32.320304 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:51:32.964444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:32.964369 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 19:51:32.964444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:32.964421 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 19:51:33.020690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.020664 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:51:33.139168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139135 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139199 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139338 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139423 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139477 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139507 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139553 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139616 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139657 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\") pod \"799befd8-3450-4228-8762-49775feb35a7\" (UID: \"799befd8-3450-4228-8762-49775feb35a7\") " Oct 30 19:51:33.139854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139718 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:33.139854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139839 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:33.139962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.139852 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:33.140089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140057 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:33.140089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140081 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:33.140309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140248 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799befd8-3450-4228-8762-49775feb35a7-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.140309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140262 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.140309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140271 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.140309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140279 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.140309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.140288 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.141618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.141594 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "kube-api-access-6g4hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:33.141787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.141766 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:33.141787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.141774 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:33.142228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.142192 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token" pod "799befd8-3450-4228-8762-49775feb35a7" (UID: "799befd8-3450-4228-8762-49775feb35a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:33.240967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.240898 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.240967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.240926 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799befd8-3450-4228-8762-49775feb35a7-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.240967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.240939 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4hx\" (UniqueName: \"kubernetes.io/projected/799befd8-3450-4228-8762-49775feb35a7-kube-api-access-6g4hx\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.240967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.240949 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799befd8-3450-4228-8762-49775feb35a7-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:33.323279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.323241 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" event={"ID":"799befd8-3450-4228-8762-49775feb35a7","Type":"ContainerDied","Data":"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f"} Oct 30 19:51:33.323279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.323273 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" Oct 30 19:51:33.323550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.323310 4789 scope.go:122] "RemoveContainer" containerID="39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f" Oct 30 19:51:33.323550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.323203 4789 generic.go:356] "Generic (PLEG): container finished" podID="799befd8-3450-4228-8762-49775feb35a7" containerID="39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f" exitCode=0 Oct 30 19:51:33.323550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.323402 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl" event={"ID":"799befd8-3450-4228-8762-49775feb35a7","Type":"ContainerDied","Data":"6ecbb02c0afaa1283303df38b34d086333802df3142b7fd9956501d5f2de363b"} Oct 30 19:51:33.336788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.336765 4789 scope.go:122] "RemoveContainer" containerID="39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f" Oct 30 19:51:33.337084 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:51:33.337065 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f\": container with ID starting with 39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f not found: ID does not exist" containerID="39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f" Oct 30 19:51:33.337159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.337096 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f"} err="failed to get container status \"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f\": rpc error: code = NotFound desc = could not find container \"39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f\": container with ID starting with 39f0f6f0850dd943b80e94ef5aa10c5083cfa6c59af2d92c5ecb76887751203f not found: ID does not exist" Oct 30 19:51:33.480450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.480383 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:51:33.484690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.484665 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4d5083-openshift-default-7f47f66745-9f2dl"] Oct 30 19:51:33.670200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.670124 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799befd8-3450-4228-8762-49775feb35a7" path="/var/lib/kubelet/pods/799befd8-3450-4228-8762-49775feb35a7/volumes" Oct 30 19:51:33.885665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.885624 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" podUID="75a4374f-abb0-41de-9a72-0befba6078b7" containerName="main" containerID="cri-o://dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd" gracePeriod=45 Oct 30 19:51:33.969408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.969381 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:51:33.970508 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:33.970488 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:51:34.165076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.165050 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:51:34.331727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.331643 4789 generic.go:356] "Generic (PLEG): container finished" podID="75a4374f-abb0-41de-9a72-0befba6078b7" containerID="dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd" exitCode=0 Oct 30 19:51:34.331727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.331685 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" Oct 30 19:51:34.331937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.331721 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" event={"ID":"75a4374f-abb0-41de-9a72-0befba6078b7","Type":"ContainerDied","Data":"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd"} Oct 30 19:51:34.331937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.331771 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv" event={"ID":"75a4374f-abb0-41de-9a72-0befba6078b7","Type":"ContainerDied","Data":"2bd12e49b0e38206b8fbf8ede6d3f55be5e2dca8870c7be73e67b25ccd7bcebe"} Oct 30 19:51:34.331937 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.331797 4789 scope.go:122] "RemoveContainer" containerID="dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd" Oct 30 19:51:34.342785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.342766 4789 scope.go:122] "RemoveContainer" containerID="dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd" Oct 30 19:51:34.343098 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:51:34.343079 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd\": container with ID starting with dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd not found: ID does not exist" containerID="dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd" Oct 30 19:51:34.343151 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.343105 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd"} err="failed to get container status \"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd\": rpc error: code = NotFound desc = could not find container \"dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd\": container with ID starting with dd425745ec3c44bcc28d48e8701911d0163394ccdba9259f7848e34720468ffd not found: ID does not exist" Oct 30 19:51:34.352535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.352507 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.352666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.352559 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.352666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.352634 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.353026 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353003 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\" (UniqueName: \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.353139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353101 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.353139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353134 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\") pod \"75a4374f-abb0-41de-9a72-0befba6078b7\" (UID: \"75a4374f-abb0-41de-9a72-0befba6078b7\") " Oct 30 19:51:34.353351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.352801 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:34.353429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353386 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:34.353663 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353627 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:34.353981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.353957 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.354140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.354124 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.354263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.354251 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.355788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.355030 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:34.355788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.355070 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:34.355788 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.355417 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz" pod "75a4374f-abb0-41de-9a72-0befba6078b7" (UID: "75a4374f-abb0-41de-9a72-0befba6078b7"). InnerVolumeSpecName "kube-api-access-n4svz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:34.455938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.455904 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a4374f-abb0-41de-9a72-0befba6078b7-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.456081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.455941 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4svz\" (UniqueName: \"kubernetes.io/projected/75a4374f-abb0-41de-9a72-0befba6078b7-kube-api-access-n4svz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.456081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.455958 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a4374f-abb0-41de-9a72-0befba6078b7-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:34.746080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.746048 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:51:34.750102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:34.750072 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-59594f/auth-test-model-59594f-kserve-7b697ff5b5-pfvsv"] Oct 30 19:51:35.670850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:35.670817 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a4374f-abb0-41de-9a72-0befba6078b7" path="/var/lib/kubelet/pods/75a4374f-abb0-41de-9a72-0befba6078b7/volumes" Oct 30 19:51:42.550502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.550464 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" podUID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" containerName="main" containerID="cri-o://40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb" gracePeriod=45 Oct 30 19:51:42.818466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.818398 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:42.836076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836050 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836108 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836154 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836172 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836246 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836280 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\" (UniqueName: \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\") pod \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\" (UID: \"48e6c818-b292-49a1-b6a2-c430ad1dc0b7\") " Oct 30 19:51:42.836498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836328 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:42.836619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836530 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:42.836737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836715 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:42.836737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836729 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:42.836852 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.836743 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:42.838487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.838449 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:42.838604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.838558 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:42.838675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.838624 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c" pod "48e6c818-b292-49a1-b6a2-c430ad1dc0b7" (UID: "48e6c818-b292-49a1-b6a2-c430ad1dc0b7"). InnerVolumeSpecName "kube-api-access-tpv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:42.937550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.937517 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:42.937550 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.937549 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:42.937782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.937563 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:42.937782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:42.937577 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpv2c\" (UniqueName: \"kubernetes.io/projected/48e6c818-b292-49a1-b6a2-c430ad1dc0b7-kube-api-access-tpv2c\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:43.395393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.395359 4789 generic.go:356] "Generic (PLEG): container finished" podID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" containerID="40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb" exitCode=0 Oct 30 19:51:43.395620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.395428 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" Oct 30 19:51:43.395620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.395458 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" event={"ID":"48e6c818-b292-49a1-b6a2-c430ad1dc0b7","Type":"ContainerDied","Data":"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb"} Oct 30 19:51:43.395620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.395503 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7" event={"ID":"48e6c818-b292-49a1-b6a2-c430ad1dc0b7","Type":"ContainerDied","Data":"e9ed3922e5ac7f7c464090583624d43847d117e02a5f8dd3a33fd7dbb8712f28"} Oct 30 19:51:43.395620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.395525 4789 scope.go:122] "RemoveContainer" containerID="40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb" Oct 30 19:51:43.406262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.406121 4789 scope.go:122] "RemoveContainer" containerID="40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb" Oct 30 19:51:43.406549 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:51:43.406519 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb\": container with ID starting with 40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb not found: ID does not exist" containerID="40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb" Oct 30 19:51:43.406635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.406560 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb"} err="failed to get container status \"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb\": rpc error: code = NotFound desc = could not find container \"40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb\": container with ID starting with 40b0f2d9283dc333fcf7c15587432d1c65daccc9e42a001f402f23c3d67559fb not found: ID does not exist" Oct 30 19:51:43.551060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.551028 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:43.599044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.599013 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4d5083/noten-model-4d5083-kserve-855bd849cc-9dlr7"] Oct 30 19:51:43.669580 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:51:43.669516 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" path="/var/lib/kubelet/pods/48e6c818-b292-49a1-b6a2-c430ad1dc0b7/volumes" Oct 30 19:51:48.196615 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:51:48.196576 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:52:10.050925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:10.050892 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5"] Oct 30 19:52:10.096879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:10.096845 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33764595-w2vk5"] Oct 30 19:52:11.668104 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:11.668069 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12233c2a-55ed-49d6-a9fd-0a676592b459" path="/var/lib/kubelet/pods/12233c2a-55ed-49d6-a9fd-0a676592b459/volumes" Oct 30 19:52:13.806546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.806511 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:52:13.807101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807086 4789 cpu_manager.go:414] "containerMap: removing container" podUID="75a4374f-abb0-41de-9a72-0befba6078b7" containerName="main" Oct 30 19:52:13.807143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807103 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a4374f-abb0-41de-9a72-0befba6078b7" containerName="main" Oct 30 19:52:13.807143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807126 4789 cpu_manager.go:414] "containerMap: removing container" podUID="799befd8-3450-4228-8762-49775feb35a7" containerName="istio-proxy" Oct 30 19:52:13.807143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807131 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799befd8-3450-4228-8762-49775feb35a7" containerName="istio-proxy" Oct 30 19:52:13.807273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807144 4789 cpu_manager.go:414] "containerMap: removing container" podUID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" containerName="main" Oct 30 19:52:13.807273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807149 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" containerName="main" Oct 30 19:52:13.807273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807262 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="799befd8-3450-4228-8762-49775feb35a7" containerName="istio-proxy" Oct 30 19:52:13.807369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807274 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e6c818-b292-49a1-b6a2-c430ad1dc0b7" containerName="main" Oct 30 19:52:13.807369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.807287 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a4374f-abb0-41de-9a72-0befba6078b7" containerName="main" Oct 30 19:52:13.811066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.811045 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.860403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.860369 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-7b6aee-openshift-default-dockercfg-n2tpq\"" Oct 30 19:52:13.918931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.918891 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:52:13.964385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964342 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964404 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964510 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964581 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964616 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964664 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964691 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964780 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:13.964971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:13.964823 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv28l\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066370 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066411 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv28l\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066542 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066597 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066644 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066703 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066735 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066784 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.066867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066808 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.067082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.066875 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.067082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.067010 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.067202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.067169 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.067537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.067474 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.067668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.067621 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.069370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.069347 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.069476 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.069399 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.138371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.138337 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv28l\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.139819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.139792 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.425742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.425650 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:14.859952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.859922 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:52:14.861423 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:14.861382 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefac3a37_19a3_4005_93e1_4496391c982d.slice/crio-364462666ebcf3c83fc400026b4a6da576f31640a058eab8ad938f86707a3da1 WatchSource:0}: Error finding container 364462666ebcf3c83fc400026b4a6da576f31640a058eab8ad938f86707a3da1: Status 404 returned error can't find the container with id 364462666ebcf3c83fc400026b4a6da576f31640a058eab8ad938f86707a3da1 Oct 30 19:52:14.864228 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.864178 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:14.864339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.864299 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:14.864409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.864346 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:14.907733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.907701 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:52:14.914484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.914451 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:52:14.914628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.914584 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:14.919730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.919709 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:14.965846 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.965820 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b330be-openshift-default-dockercfg-w965c\"" Oct 30 19:52:14.975886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:14.975856 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:52:15.048183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.048147 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078582 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078625 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078644 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078661 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078694 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078713 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8cd\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078732 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpdw\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078762 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078780 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078813 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078830 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078852 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078875 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078901 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078924 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078951 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078969 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.081245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.078984 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180112 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180156 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180179 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180233 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180273 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180302 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8cd\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180332 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpdw\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180375 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180401 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180459 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180485 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180526 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180567 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180636 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.180695 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180653 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.181075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180702 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.181075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180731 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.181075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.180758 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.181372 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.181398 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.181544 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.181795 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.181923 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.182042 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.182261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.182047 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.182624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.182597 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.182816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.182797 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.183673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.183148 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.183673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.183632 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.183931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.183908 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.185144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.185118 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.238907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.238870 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.239159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.239133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpdw\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.239706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.239681 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8cd\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.239801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.239718 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\") pod \"e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.249717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.249682 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:15.256999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.256970 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:15.638417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.638384 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerStarted","Data":"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2"} Oct 30 19:52:15.638641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.638445 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerStarted","Data":"364462666ebcf3c83fc400026b4a6da576f31640a058eab8ad938f86707a3da1"} Oct 30 19:52:15.689978 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:15.689893 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a422e4_d688_468b_b210_44aa1f347e42.slice/crio-0d18bcc0d6d7fa5a068299526f83ed1b0eb637cc1416f162bca7f30a37db3064 WatchSource:0}: Error finding container 0d18bcc0d6d7fa5a068299526f83ed1b0eb637cc1416f162bca7f30a37db3064: Status 404 returned error can't find the container with id 0d18bcc0d6d7fa5a068299526f83ed1b0eb637cc1416f162bca7f30a37db3064 Oct 30 19:52:15.690175 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:15.690153 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d49601_86c3_41d4_a922_68f4b660d039.slice/crio-832de4f771aac3d5f5b4257589da7c90b45d2a43179cf38c4a95dbe8efab58d0 WatchSource:0}: Error finding container 832de4f771aac3d5f5b4257589da7c90b45d2a43179cf38c4a95dbe8efab58d0: Status 404 returned error can't find the container with id 832de4f771aac3d5f5b4257589da7c90b45d2a43179cf38c4a95dbe8efab58d0 Oct 30 19:52:15.699818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.699785 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:52:15.699952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.699893 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:52:15.703732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.703693 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.703821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.703775 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.703821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.703811 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.704495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.704460 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.704587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.704533 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.704587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.704573 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:15.801537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:15.801478 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podStartSLOduration=2.801461996 podStartE2EDuration="2.801461996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:15.798799435 +0000 UTC m=+3972.300709921" watchObservedRunningTime="2026-10-30 19:52:15.801461996 +0000 UTC m=+3972.303372481" Oct 30 19:52:16.425864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.425805 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:16.433986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.433952 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:16.658592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.658543 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" event={"ID":"d2a422e4-d688-468b-b210-44aa1f347e42","Type":"ContainerStarted","Data":"cd472d212658a1bdb3413597114d0945efe4053f3cbc6a3571cad54a4b7f7670"} Oct 30 19:52:16.658804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.658619 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" event={"ID":"d2a422e4-d688-468b-b210-44aa1f347e42","Type":"ContainerStarted","Data":"0d18bcc0d6d7fa5a068299526f83ed1b0eb637cc1416f162bca7f30a37db3064"} Oct 30 19:52:16.662243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.662195 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" event={"ID":"42d49601-86c3-41d4-a922-68f4b660d039","Type":"ContainerStarted","Data":"1e9924635cecef3700c4525e4d77f196703d8753bb278040af84ba21c171de4b"} Oct 30 19:52:16.662397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.662257 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" event={"ID":"42d49601-86c3-41d4-a922-68f4b660d039","Type":"ContainerStarted","Data":"832de4f771aac3d5f5b4257589da7c90b45d2a43179cf38c4a95dbe8efab58d0"} Oct 30 19:52:16.662717 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.662698 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:16.663874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.663855 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:52:16.780357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:16.780293 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podStartSLOduration=2.780276745 podStartE2EDuration="2.780276745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:16.777474318 +0000 UTC m=+3973.279384806" watchObservedRunningTime="2026-10-30 19:52:16.780276745 +0000 UTC m=+3973.282187231" Oct 30 19:52:17.027281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.027195 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podStartSLOduration=3.027176569 podStartE2EDuration="3.027176569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:16.903923248 +0000 UTC m=+3973.405833739" watchObservedRunningTime="2026-10-30 19:52:17.027176569 +0000 UTC m=+3973.529087056" Oct 30 19:52:17.255024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.254989 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:17.259314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.258381 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.259314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.258444 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 19:52:17.259314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.258638 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:17.260086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.260055 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 19:52:17.260196 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:17.260104 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 19:52:18.250933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:18.250513 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.250933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:18.250572 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 19:52:18.257622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:18.257513 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 19:52:18.257622 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:18.257561 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 19:52:18.263373 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:52:18.263284 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:52:19.250665 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:19.250631 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.250842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:19.250681 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 19:52:19.257995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:19.257971 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 19:52:19.258431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:19.258409 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 19:52:20.250753 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:20.250715 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.250999 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:20.250973 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 19:52:20.257498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:20.257470 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 19:52:20.257714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:20.257689 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 19:52:21.059500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.059465 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb"] Oct 30 19:52:21.110835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.110794 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf3c95be-tbprb"] Oct 30 19:52:21.250862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.250816 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.251085 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.250875 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 30 19:52:21.257552 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.257513 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Oct 30 19:52:21.257710 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.257565 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Oct 30 19:52:21.670203 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:21.670168 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da61215e-da45-4fd6-88b9-892e00d91696" path="/var/lib/kubelet/pods/da61215e-da45-4fd6-88b9-892e00d91696/volumes" Oct 30 19:52:22.254505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.254474 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:22.254995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.254968 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:22.255871 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.255851 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:52:22.262268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.262246 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:22.262631 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.262608 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:22.263337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:22.263320 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:52:23.770970 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.770941 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:52:23.776968 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.776948 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.794141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.794109 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2h9m\" (UniqueName: \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.794287 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.794260 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.826081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.825948 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-7b6aee\"" Oct 30 19:52:23.907280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.887966 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-7b6aee-dockercfg-t2kpg\"" Oct 30 19:52:23.907280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.895685 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2h9m\" (UniqueName: \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.907280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.895764 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.907280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.896817 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.907280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.902261 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:52:23.933044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.933010 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:52:23.937809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.937789 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:23.954881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.954844 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2h9m\" (UniqueName: \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\") pod \"payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:23.993731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:23.993699 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:52:24.004038 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.004005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.004201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.004124 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghjr\" (UniqueName: \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.047874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.047805 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649"] Oct 30 19:52:24.054284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.054255 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a19647eb-fx649"] Oct 30 19:52:24.091608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.091581 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:24.105754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.105729 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghjr\" (UniqueName: \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.105886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.105866 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.106593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.106572 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.171529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.171500 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghjr\" (UniqueName: \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\") pod \"payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.251705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.251676 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.464358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.464331 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:52:24.465648 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:24.465620 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe7d04c_da06_4e27_9630_d0eb2c0c0769.slice/crio-f0b5fe307e96d112a8d51e0ca143faa9ee4678a3e65add8a9dbbd21ab7f45294 WatchSource:0}: Error finding container f0b5fe307e96d112a8d51e0ca143faa9ee4678a3e65add8a9dbbd21ab7f45294: Status 404 returned error can't find the container with id f0b5fe307e96d112a8d51e0ca143faa9ee4678a3e65add8a9dbbd21ab7f45294 Oct 30 19:52:24.629268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.629245 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:52:24.630536 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:24.630501 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26cd257a_812f_416f_842c_a6f67fbf290d.slice/crio-5160ba439c75d690ef20c2ab069befe285db5d872954ccfbf2c0a80ce41ee34d WatchSource:0}: Error finding container 5160ba439c75d690ef20c2ab069befe285db5d872954ccfbf2c0a80ce41ee34d: Status 404 returned error can't find the container with id 5160ba439c75d690ef20c2ab069befe285db5d872954ccfbf2c0a80ce41ee34d Oct 30 19:52:24.767346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.767315 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" event={"ID":"26cd257a-812f-416f-842c-a6f67fbf290d","Type":"ContainerStarted","Data":"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b"} Oct 30 19:52:24.767507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.767378 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" event={"ID":"26cd257a-812f-416f-842c-a6f67fbf290d","Type":"ContainerStarted","Data":"5160ba439c75d690ef20c2ab069befe285db5d872954ccfbf2c0a80ce41ee34d"} Oct 30 19:52:24.767582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.767502 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:24.769091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.769064 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" event={"ID":"afe7d04c-da06-4e27-9630-d0eb2c0c0769","Type":"ContainerStarted","Data":"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017"} Oct 30 19:52:24.769200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.769108 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" event={"ID":"afe7d04c-da06-4e27-9630-d0eb2c0c0769","Type":"ContainerStarted","Data":"f0b5fe307e96d112a8d51e0ca143faa9ee4678a3e65add8a9dbbd21ab7f45294"} Oct 30 19:52:24.769200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.769173 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:24.878668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.878588 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" podStartSLOduration=1.87857648 podStartE2EDuration="1.87857648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:24.875405016 +0000 UTC m=+3981.377315499" watchObservedRunningTime="2026-10-30 19:52:24.87857648 +0000 UTC m=+3981.380486965" Oct 30 19:52:24.998199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:24.998117 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" podStartSLOduration=1.998098363 podStartE2EDuration="1.998098363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:24.997852363 +0000 UTC m=+3981.499762850" watchObservedRunningTime="2026-10-30 19:52:24.998098363 +0000 UTC m=+3981.500008850" Oct 30 19:52:25.720234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:25.720085 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5198cc1-9605-40e6-bea1-183f2cdabdbd" path="/var/lib/kubelet/pods/f5198cc1-9605-40e6-bea1-183f2cdabdbd/volumes" Oct 30 19:52:33.017688 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.017651 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:52:33.023018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.023000 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.072762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.072721 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-312161\"" Oct 30 19:52:33.144267 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.113886 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.144617 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.125433 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-312161-dockercfg-764fk\"" Oct 30 19:52:33.144721 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.127935 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:52:33.145693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.145575 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbbs\" (UniqueName: \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.197064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.197034 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:52:33.201355 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.201339 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.248078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.248051 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbbs\" (UniqueName: \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.248247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.248108 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.248794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.248774 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.256256 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.256229 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:52:33.306499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.306427 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbbs\" (UniqueName: \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\") pod \"payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.337136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.337112 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.349248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.349204 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6bcv\" (UniqueName: \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.349401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.349386 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.453881 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.453830 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6bcv\" (UniqueName: \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.454066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.454048 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.455022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.454992 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.513205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.513168 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6bcv\" (UniqueName: \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\") pod \"payload-processing-e2e-extmod-b-312161-6f68889998-rghph\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.518758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.518728 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:33.716176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.716147 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:52:33.718428 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:33.718398 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b084f6_bd1c_4274_bac7_be10962e4c7c.slice/crio-5239cffa21d079e4fe82fa64c8fa98ef251fa29bc1e059eca3aacb116b2c8d67 WatchSource:0}: Error finding container 5239cffa21d079e4fe82fa64c8fa98ef251fa29bc1e059eca3aacb116b2c8d67: Status 404 returned error can't find the container with id 5239cffa21d079e4fe82fa64c8fa98ef251fa29bc1e059eca3aacb116b2c8d67 Oct 30 19:52:33.838477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.838401 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" event={"ID":"12b084f6-bd1c-4274-bac7-be10962e4c7c","Type":"ContainerStarted","Data":"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12"} Oct 30 19:52:33.838477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.838463 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" event={"ID":"12b084f6-bd1c-4274-bac7-be10962e4c7c","Type":"ContainerStarted","Data":"5239cffa21d079e4fe82fa64c8fa98ef251fa29bc1e059eca3aacb116b2c8d67"} Oct 30 19:52:33.838648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.838495 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:33.896034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.896012 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:52:33.897282 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:52:33.897254 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc329f6_8646_4186_a57d_aa3b58103119.slice/crio-40671b5a72ef64708e586ecdff21a36a16a09b31653eeffc6952fb14c48556ed WatchSource:0}: Error finding container 40671b5a72ef64708e586ecdff21a36a16a09b31653eeffc6952fb14c48556ed: Status 404 returned error can't find the container with id 40671b5a72ef64708e586ecdff21a36a16a09b31653eeffc6952fb14c48556ed Oct 30 19:52:33.951487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:33.951434 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" podStartSLOduration=1.951420663 podStartE2EDuration="1.951420663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:33.948277659 +0000 UTC m=+3990.450188159" watchObservedRunningTime="2026-10-30 19:52:33.951420663 +0000 UTC m=+3990.453331148" Oct 30 19:52:34.849411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:34.849373 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" event={"ID":"edc329f6-8646-4186-a57d-aa3b58103119","Type":"ContainerStarted","Data":"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63"} Oct 30 19:52:34.849835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:34.849435 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" event={"ID":"edc329f6-8646-4186-a57d-aa3b58103119","Type":"ContainerStarted","Data":"40671b5a72ef64708e586ecdff21a36a16a09b31653eeffc6952fb14c48556ed"} Oct 30 19:52:34.849835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:34.849560 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:35.779627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:35.779599 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:52:35.779825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:35.779671 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:52:35.890158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:35.890108 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" podStartSLOduration=2.8900944859999997 podStartE2EDuration="2.890094486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:34.96212299 +0000 UTC m=+3991.464033479" watchObservedRunningTime="2026-10-30 19:52:35.890094486 +0000 UTC m=+3992.392004975" Oct 30 19:52:44.852333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:44.852261 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:52:45.860293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:52:45.860265 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:52:48.196669 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:52:48.196618 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:53:07.307860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:07.307828 4789 scope.go:122] "RemoveContainer" containerID="2b5f56d4085df1ccd964a5cb8241d869f73b530b0b5d1c620265298caa5f50b2" Oct 30 19:53:07.320576 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:07.320556 4789 scope.go:122] "RemoveContainer" containerID="710074b3ccf51260c1adc957a35f93961f81a9b0064ea102a2b34221084f809a" Oct 30 19:53:18.197845 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:53:18.197804 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:53:31.321404 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.321367 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:53:31.326293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.326269 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.382158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.382123 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:53:31.396850 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.396824 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.396994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.396900 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.396994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.396978 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.397159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.397023 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.397159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.397052 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.397289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.397256 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjkw\" (UniqueName: \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.423656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.423636 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b330be\"/\"kube-root-ca.crt\"" Oct 30 19:53:31.423813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.423794 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b330be\"/\"explicit-model-b330be-kserve-self-signed-certs\"" Oct 30 19:53:31.423880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.423818 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b330be\"/\"openshift-service-ca.crt\"" Oct 30 19:53:31.423880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.423873 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b330be\"/\"default-dockercfg-fg8kl\"" Oct 30 19:53:31.498232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498175 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjkw\" (UniqueName: \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498261 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498315 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498408 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498362 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498411 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498583 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498451 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498799 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498778 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498824 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.498877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.498847 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.500615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.500596 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.501099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.501083 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.556428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.556397 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjkw\" (UniqueName: \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\") pod \"explicit-model-b330be-kserve-847778fc-vll2v\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:31.639260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:31.639143 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:32.021892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:32.021865 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:53:32.023929 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:53:32.023898 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3284beae_ea6d_48b7_ae67_c3fab4c1552c.slice/crio-f22e37a1701df91d7c55a295a002be12af117f69489047c0c563aaff30c10ca9 WatchSource:0}: Error finding container f22e37a1701df91d7c55a295a002be12af117f69489047c0c563aaff30c10ca9: Status 404 returned error can't find the container with id f22e37a1701df91d7c55a295a002be12af117f69489047c0c563aaff30c10ca9 Oct 30 19:53:32.295243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:32.295151 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" event={"ID":"3284beae-ea6d-48b7-ae67-c3fab4c1552c","Type":"ContainerStarted","Data":"a51ffeef3bf4dc364c3729d18325f8dfdc0327daf0613754dd63648e7be97620"} Oct 30 19:53:32.295243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:32.295191 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" event={"ID":"3284beae-ea6d-48b7-ae67-c3fab4c1552c","Type":"ContainerStarted","Data":"f22e37a1701df91d7c55a295a002be12af117f69489047c0c563aaff30c10ca9"} Oct 30 19:53:32.408883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:32.408834 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" podStartSLOduration=1.408820394 podStartE2EDuration="1.408820394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:32.405178024 +0000 UTC m=+4048.907088508" watchObservedRunningTime="2026-10-30 19:53:32.408820394 +0000 UTC m=+4048.910730875" Oct 30 19:53:32.639888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:32.639816 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:34.056487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:34.056448 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7"] Oct 30 19:53:34.059392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:34.059369 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-569b6fd1-22nv7"] Oct 30 19:53:35.668629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:35.668594 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b76897d-1c1f-4301-88a3-ed80ea6e8a99" path="/var/lib/kubelet/pods/9b76897d-1c1f-4301-88a3-ed80ea6e8a99/volumes" Oct 30 19:53:41.640282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:41.640247 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:41.652577 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:41.652551 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:42.039820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.039783 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh"] Oct 30 19:53:42.043985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.043965 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.093181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.093157 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:53:42.093438 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.093189 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:53:42.093544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.093190 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:42.093633 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.093339 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:53:42.097122 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.097100 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh"] Oct 30 19:53:42.217298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.214082 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.217298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.214277 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.217298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.214354 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppb8j\" (UniqueName: \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.315044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.314964 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.315191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.315046 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppb8j\" (UniqueName: \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.315191 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.315115 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.315922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.315893 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.317433 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.317405 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.373003 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.372975 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppb8j\" (UniqueName: \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\") pod \"maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:42.382509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.382490 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:53:42.658674 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:42.658585 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:43.039462 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:43.039426 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh"] Oct 30 19:53:43.041958 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:53:43.041930 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442a3725_c4c5_4b1e_a233_0a24ba92e431.slice/crio-4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6 WatchSource:0}: Error finding container 4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6: Status 404 returned error can't find the container with id 4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6 Oct 30 19:53:43.379410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:43.379377 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" event={"ID":"442a3725-c4c5-4b1e-a233-0a24ba92e431","Type":"ContainerStarted","Data":"9c9e960d03f867d18ac1225a2dde4f80055d924dfa891596f1f26166a647cbc2"} Oct 30 19:53:43.379573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:43.379431 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" event={"ID":"442a3725-c4c5-4b1e-a233-0a24ba92e431","Type":"ContainerStarted","Data":"4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6"} Oct 30 19:53:43.489730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:43.489683 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" podStartSLOduration=2.489668776 podStartE2EDuration="2.489668776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:43.488007863 +0000 UTC m=+4059.989918349" watchObservedRunningTime="2026-10-30 19:53:43.489668776 +0000 UTC m=+4059.991579261" Oct 30 19:53:44.387499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:44.387440 4789 generic.go:356] "Generic (PLEG): container finished" podID="442a3725-c4c5-4b1e-a233-0a24ba92e431" containerID="9c9e960d03f867d18ac1225a2dde4f80055d924dfa891596f1f26166a647cbc2" exitCode=0 Oct 30 19:53:44.387913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:44.387529 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" event={"ID":"442a3725-c4c5-4b1e-a233-0a24ba92e431","Type":"ContainerDied","Data":"9c9e960d03f867d18ac1225a2dde4f80055d924dfa891596f1f26166a647cbc2"} Oct 30 19:53:45.049273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.049239 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7"] Oct 30 19:53:45.095751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.095720 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fa8d657-m4mf7"] Oct 30 19:53:45.588194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.584452 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:45.671724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.671653 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3" path="/var/lib/kubelet/pods/bdf59d71-664d-44d1-8ff5-99c1fdb8a2f3/volumes" Oct 30 19:53:45.758644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.758610 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\" (UniqueName: \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\") pod \"442a3725-c4c5-4b1e-a233-0a24ba92e431\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " Oct 30 19:53:45.758815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.758682 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\") pod \"442a3725-c4c5-4b1e-a233-0a24ba92e431\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " Oct 30 19:53:45.758815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.758707 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\") pod \"442a3725-c4c5-4b1e-a233-0a24ba92e431\" (UID: \"442a3725-c4c5-4b1e-a233-0a24ba92e431\") " Oct 30 19:53:45.759310 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.759280 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca" pod "442a3725-c4c5-4b1e-a233-0a24ba92e431" (UID: "442a3725-c4c5-4b1e-a233-0a24ba92e431"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:45.765393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.765364 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca" pod "442a3725-c4c5-4b1e-a233-0a24ba92e431" (UID: "442a3725-c4c5-4b1e-a233-0a24ba92e431"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:45.765505 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.765401 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j" pod "442a3725-c4c5-4b1e-a233-0a24ba92e431" (UID: "442a3725-c4c5-4b1e-a233-0a24ba92e431"). InnerVolumeSpecName "kube-api-access-ppb8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:45.859813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.859786 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppb8j\" (UniqueName: \"kubernetes.io/projected/442a3725-c4c5-4b1e-a233-0a24ba92e431-kube-api-access-ppb8j\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:45.859813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.859812 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:45.859975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:45.859822 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/442a3725-c4c5-4b1e-a233-0a24ba92e431-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:46.406899 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:46.406854 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" event={"ID":"442a3725-c4c5-4b1e-a233-0a24ba92e431","Type":"ContainerDied","Data":"4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6"} Oct 30 19:53:46.407076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:46.406952 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e45ed04b6c3a331c0c2abf825128add0a67b4afb06cbdf5e77d31ddbf19d2d6" Oct 30 19:53:46.407076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:46.406870 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh" Oct 30 19:53:48.196164 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:53:48.196128 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:53:53.433013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:53.432976 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:53:53.535314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:53.535278 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:53:58.433795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.433756 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" podUID="db28d4c9-9cd1-4d28-8393-15718836648a" containerName="payload-processing" containerID="cri-o://43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee" gracePeriod=25 Oct 30 19:53:58.536086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.536046 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" podUID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" containerName="payload-pre-processing" containerID="cri-o://3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e" gracePeriod=25 Oct 30 19:53:58.698064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.697838 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:53:58.799192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.799167 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\") pod \"db28d4c9-9cd1-4d28-8393-15718836648a\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " Oct 30 19:53:58.799321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.799229 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\" (UniqueName: \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\") pod \"db28d4c9-9cd1-4d28-8393-15718836648a\" (UID: \"db28d4c9-9cd1-4d28-8393-15718836648a\") " Oct 30 19:53:58.799611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.799577 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume" pod "db28d4c9-9cd1-4d28-8393-15718836648a" (UID: "db28d4c9-9cd1-4d28-8393-15718836648a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:58.801334 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.801312 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t" pod "db28d4c9-9cd1-4d28-8393-15718836648a" (UID: "db28d4c9-9cd1-4d28-8393-15718836648a"). InnerVolumeSpecName "kube-api-access-4r24t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:58.815002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.814978 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:53:58.900499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.900473 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\" (UniqueName: \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\") pod \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " Oct 30 19:53:58.900641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.900529 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\") pod \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\" (UID: \"f4f19999-bc7b-401c-8cf1-81b65f6697dd\") " Oct 30 19:53:58.900962 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.900930 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume" pod "f4f19999-bc7b-401c-8cf1-81b65f6697dd" (UID: "f4f19999-bc7b-401c-8cf1-81b65f6697dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:58.901069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.900995 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db28d4c9-9cd1-4d28-8393-15718836648a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.901069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.901009 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r24t\" (UniqueName: \"kubernetes.io/projected/db28d4c9-9cd1-4d28-8393-15718836648a-kube-api-access-4r24t\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.902626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:58.902604 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr" pod "f4f19999-bc7b-401c-8cf1-81b65f6697dd" (UID: "f4f19999-bc7b-401c-8cf1-81b65f6697dd"). InnerVolumeSpecName "kube-api-access-7tvrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:59.001733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.001669 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvrr\" (UniqueName: \"kubernetes.io/projected/f4f19999-bc7b-401c-8cf1-81b65f6697dd-kube-api-access-7tvrr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:59.001733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.001697 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f19999-bc7b-401c-8cf1-81b65f6697dd-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:59.501027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.500984 4789 generic.go:356] "Generic (PLEG): container finished" podID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" containerID="3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e" exitCode=0 Oct 30 19:53:59.501484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.501046 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" Oct 30 19:53:59.501484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.501068 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" event={"ID":"f4f19999-bc7b-401c-8cf1-81b65f6697dd","Type":"ContainerDied","Data":"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e"} Oct 30 19:53:59.501484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.501148 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx" event={"ID":"f4f19999-bc7b-401c-8cf1-81b65f6697dd","Type":"ContainerDied","Data":"1ae5f2acf23b10ddc699bd95d794b00cd3b7b279f89378e64e7043a491800068"} Oct 30 19:53:59.501484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.501179 4789 scope.go:122] "RemoveContainer" containerID="3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e" Oct 30 19:53:59.503249 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.503172 4789 generic.go:356] "Generic (PLEG): container finished" podID="db28d4c9-9cd1-4d28-8393-15718836648a" containerID="43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee" exitCode=0 Oct 30 19:53:59.503347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.503274 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" event={"ID":"db28d4c9-9cd1-4d28-8393-15718836648a","Type":"ContainerDied","Data":"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee"} Oct 30 19:53:59.503347 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.503314 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" event={"ID":"db28d4c9-9cd1-4d28-8393-15718836648a","Type":"ContainerDied","Data":"dfc2a04ddc292be406e9f60400f252d6dff4da0b2fe5249b85371ade8cef8b8b"} Oct 30 19:53:59.503434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.503395 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd" Oct 30 19:53:59.518529 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.518293 4789 scope.go:122] "RemoveContainer" containerID="3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e" Oct 30 19:53:59.518777 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:53:59.518634 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e\": container with ID starting with 3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e not found: ID does not exist" containerID="3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e" Oct 30 19:53:59.518777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.518671 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e"} err="failed to get container status \"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e\": rpc error: code = NotFound desc = could not find container \"3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e\": container with ID starting with 3244eee702ab88aaf70fc777d599c9524f51d26c3e3b5027af0750c5c68e447e not found: ID does not exist" Oct 30 19:53:59.518777 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.518695 4789 scope.go:122] "RemoveContainer" containerID="43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee" Oct 30 19:53:59.528933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.528914 4789 scope.go:122] "RemoveContainer" containerID="43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee" Oct 30 19:53:59.529237 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:53:59.529183 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee\": container with ID starting with 43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee not found: ID does not exist" containerID="43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee" Oct 30 19:53:59.529377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.529246 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee"} err="failed to get container status \"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee\": rpc error: code = NotFound desc = could not find container \"43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee\": container with ID starting with 43b14cc787fb94172af39009e10ad456a2fca22da27fe5d59e2f990be8d07cee not found: ID does not exist" Oct 30 19:53:59.670836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.670805 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:53:59.682477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.682448 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-17216f-6f7f48579d-2d5gd"] Oct 30 19:53:59.801029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.800948 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:53:59.835528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:53:59.835500 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-17216f-86b7695579-c76xx"] Oct 30 19:54:01.671494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:01.671457 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db28d4c9-9cd1-4d28-8393-15718836648a" path="/var/lib/kubelet/pods/db28d4c9-9cd1-4d28-8393-15718836648a/volumes" Oct 30 19:54:01.672028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:01.672010 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" path="/var/lib/kubelet/pods/f4f19999-bc7b-401c-8cf1-81b65f6697dd/volumes" Oct 30 19:54:07.354383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:07.354351 4789 scope.go:122] "RemoveContainer" containerID="ea453548df83c2ed7b6b41a0914e73aba9b0b5b950155fa2eb198ce5ea7b237b" Oct 30 19:54:07.363870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:07.363847 4789 scope.go:122] "RemoveContainer" containerID="0b2d322e1c03a48b76cbbfa8e66cbf1870060a24f9f96059014134d06a7395fe" Oct 30 19:54:11.394471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.394437 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2"] Oct 30 19:54:11.395014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395001 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" containerName="payload-pre-processing" Oct 30 19:54:11.395062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395015 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" containerName="payload-pre-processing" Oct 30 19:54:11.395062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395047 4789 cpu_manager.go:414] "containerMap: removing container" podUID="db28d4c9-9cd1-4d28-8393-15718836648a" containerName="payload-processing" Oct 30 19:54:11.395062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395053 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db28d4c9-9cd1-4d28-8393-15718836648a" containerName="payload-processing" Oct 30 19:54:11.395166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395071 4789 cpu_manager.go:414] "containerMap: removing container" podUID="442a3725-c4c5-4b1e-a233-0a24ba92e431" containerName="revoke-keys" Oct 30 19:54:11.395166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395076 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442a3725-c4c5-4b1e-a233-0a24ba92e431" containerName="revoke-keys" Oct 30 19:54:11.395262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395167 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f19999-bc7b-401c-8cf1-81b65f6697dd" containerName="payload-pre-processing" Oct 30 19:54:11.395262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395179 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="db28d4c9-9cd1-4d28-8393-15718836648a" containerName="payload-processing" Oct 30 19:54:11.395262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.395189 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="442a3725-c4c5-4b1e-a233-0a24ba92e431" containerName="revoke-keys" Oct 30 19:54:11.398471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.398454 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.449469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.449448 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:54:11.451981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.451962 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:11.498352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.498324 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:54:11.499333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.499309 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2"] Oct 30 19:54:11.533877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.533847 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfs8\" (UniqueName: \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.533997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.533981 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.534075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.534059 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.549103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.549084 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:54:11.634837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.634813 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.634949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.634887 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfs8\" (UniqueName: \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.634998 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.634979 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.635640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.635618 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.637155 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.637133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.695420 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.695401 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfs8\" (UniqueName: \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\") pod \"maas-api-revoke-keys-e2e-explicit-b330be-nv9q2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:11.711801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:11.711782 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:12.083982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:12.083957 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2"] Oct 30 19:54:12.085942 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:54:12.085915 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8232f2_a08c_487e_9cec_a752da7594d2.slice/crio-d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7 WatchSource:0}: Error finding container d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7: Status 404 returned error can't find the container with id d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7 Oct 30 19:54:12.598544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:12.598492 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" event={"ID":"bd8232f2-a08c-487e-9cec-a752da7594d2","Type":"ContainerStarted","Data":"0c8e5016a6f901d7d1362a3cc58343eb699d9eb6d8d2439b8b494f9cacbba127"} Oct 30 19:54:12.599016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:12.598564 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" event={"ID":"bd8232f2-a08c-487e-9cec-a752da7594d2","Type":"ContainerStarted","Data":"d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7"} Oct 30 19:54:12.708011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:12.707961 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" podStartSLOduration=1.7079482129999999 podStartE2EDuration="1.707948213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:12.707550717 +0000 UTC m=+4089.209461204" watchObservedRunningTime="2026-10-30 19:54:12.707948213 +0000 UTC m=+4089.209858699" Oct 30 19:54:13.609292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:13.609257 4789 generic.go:356] "Generic (PLEG): container finished" podID="bd8232f2-a08c-487e-9cec-a752da7594d2" containerID="0c8e5016a6f901d7d1362a3cc58343eb699d9eb6d8d2439b8b494f9cacbba127" exitCode=0 Oct 30 19:54:13.609697 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:13.609327 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" event={"ID":"bd8232f2-a08c-487e-9cec-a752da7594d2","Type":"ContainerDied","Data":"0c8e5016a6f901d7d1362a3cc58343eb699d9eb6d8d2439b8b494f9cacbba127"} Oct 30 19:54:14.783974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.783942 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:14.869125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.869051 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\") pod \"bd8232f2-a08c-487e-9cec-a752da7594d2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " Oct 30 19:54:14.869125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.869102 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\" (UniqueName: \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\") pod \"bd8232f2-a08c-487e-9cec-a752da7594d2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " Oct 30 19:54:14.869329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.869189 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\") pod \"bd8232f2-a08c-487e-9cec-a752da7594d2\" (UID: \"bd8232f2-a08c-487e-9cec-a752da7594d2\") " Oct 30 19:54:14.869454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.869425 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca" pod "bd8232f2-a08c-487e-9cec-a752da7594d2" (UID: "bd8232f2-a08c-487e-9cec-a752da7594d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:14.869666 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.869645 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:14.871169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.871144 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca" pod "bd8232f2-a08c-487e-9cec-a752da7594d2" (UID: "bd8232f2-a08c-487e-9cec-a752da7594d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:14.871285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.871203 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8" pod "bd8232f2-a08c-487e-9cec-a752da7594d2" (UID: "bd8232f2-a08c-487e-9cec-a752da7594d2"). InnerVolumeSpecName "kube-api-access-psfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:14.971013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.970983 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8232f2-a08c-487e-9cec-a752da7594d2-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:14.971013 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:14.971008 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psfs8\" (UniqueName: \"kubernetes.io/projected/bd8232f2-a08c-487e-9cec-a752da7594d2-kube-api-access-psfs8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:15.625237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:15.625179 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" Oct 30 19:54:15.625237 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:15.625205 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2" event={"ID":"bd8232f2-a08c-487e-9cec-a752da7594d2","Type":"ContainerDied","Data":"d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7"} Oct 30 19:54:15.625498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:15.625269 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73da2f15078256a14f1d55c82886fd0ea8c08a64adcebf1675c5f0983c6d9d7" Oct 30 19:54:18.197552 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:54:18.197504 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:54:25.949440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:25.949407 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx"] Oct 30 19:54:25.949979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:25.949964 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bd8232f2-a08c-487e-9cec-a752da7594d2" containerName="revoke-keys" Oct 30 19:54:25.950031 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:25.949980 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8232f2-a08c-487e-9cec-a752da7594d2" containerName="revoke-keys" Oct 30 19:54:25.950111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:25.950101 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8232f2-a08c-487e-9cec-a752da7594d2" containerName="revoke-keys" Oct 30 19:54:25.953804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:25.953788 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.014232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.007672 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:54:26.044392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.044364 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.044606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.044591 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.044714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.044703 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr8b2\" (UniqueName: \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.053751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.053467 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:54:26.053751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.053506 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:26.054002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.053986 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:54:26.071458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.071425 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx"] Oct 30 19:54:26.145685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.145656 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr8b2\" (UniqueName: \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.145992 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.145976 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.146119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.146108 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.146732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.146701 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.148545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.148519 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.220274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.216925 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr8b2\" (UniqueName: \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\") pod \"maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.268090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.268064 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:26.682933 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.682831 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx"] Oct 30 19:54:26.688011 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:54:26.687981 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56095386_d7f2_4ad9_a2cf_382a264b67fa.slice/crio-d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60 WatchSource:0}: Error finding container d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60: Status 404 returned error can't find the container with id d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60 Oct 30 19:54:26.736331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:26.736284 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" event={"ID":"56095386-d7f2-4ad9-a2cf-382a264b67fa","Type":"ContainerStarted","Data":"d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60"} Oct 30 19:54:27.745794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:27.745751 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" event={"ID":"56095386-d7f2-4ad9-a2cf-382a264b67fa","Type":"ContainerStarted","Data":"dfd5b60edf83667dc4e8aa5a6c8fb181edafb864dc7c95e6eea9857c05031c60"} Oct 30 19:54:27.861194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:27.861137 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" podStartSLOduration=2.861121645 podStartE2EDuration="2.861121645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:27.858553133 +0000 UTC m=+4104.360463620" watchObservedRunningTime="2026-10-30 19:54:27.861121645 +0000 UTC m=+4104.363032130" Oct 30 19:54:28.757534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:28.757498 4789 generic.go:356] "Generic (PLEG): container finished" podID="56095386-d7f2-4ad9-a2cf-382a264b67fa" containerID="dfd5b60edf83667dc4e8aa5a6c8fb181edafb864dc7c95e6eea9857c05031c60" exitCode=0 Oct 30 19:54:28.757951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:28.757587 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" event={"ID":"56095386-d7f2-4ad9-a2cf-382a264b67fa","Type":"ContainerDied","Data":"dfd5b60edf83667dc4e8aa5a6c8fb181edafb864dc7c95e6eea9857c05031c60"} Oct 30 19:54:29.766168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:29.766131 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:54:30.006217 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.006173 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:30.026969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.026899 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:54:30.028924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.028896 4789 cpu_manager.go:414] "containerMap: removing container" podUID="56095386-d7f2-4ad9-a2cf-382a264b67fa" containerName="revoke-keys" Oct 30 19:54:30.028924 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.028924 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56095386-d7f2-4ad9-a2cf-382a264b67fa" containerName="revoke-keys" Oct 30 19:54:30.029108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.029096 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="56095386-d7f2-4ad9-a2cf-382a264b67fa" containerName="revoke-keys" Oct 30 19:54:30.033111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.033092 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.124385 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.124351 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:54:30.137696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.137670 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\") pod \"56095386-d7f2-4ad9-a2cf-382a264b67fa\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " Oct 30 19:54:30.138124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.138101 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\") pod \"56095386-d7f2-4ad9-a2cf-382a264b67fa\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " Oct 30 19:54:30.138263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.138186 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\" (UniqueName: \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\") pod \"56095386-d7f2-4ad9-a2cf-382a264b67fa\" (UID: \"56095386-d7f2-4ad9-a2cf-382a264b67fa\") " Oct 30 19:54:30.138591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.138425 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85lr7\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.138828 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.138519 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca" pod "56095386-d7f2-4ad9-a2cf-382a264b67fa" (UID: "56095386-d7f2-4ad9-a2cf-382a264b67fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:30.138957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.138856 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139081 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139140 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139263 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139237 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139335 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139366 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139427 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139399 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139471 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139460 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.139553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.139538 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:30.142247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.141334 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca" pod "56095386-d7f2-4ad9-a2cf-382a264b67fa" (UID: "56095386-d7f2-4ad9-a2cf-382a264b67fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:30.142683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.142657 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2" pod "56095386-d7f2-4ad9-a2cf-382a264b67fa" (UID: "56095386-d7f2-4ad9-a2cf-382a264b67fa"). InnerVolumeSpecName "kube-api-access-lr8b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:30.240555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240523 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.240555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240558 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.240794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240579 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.240794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240778 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.240913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240856 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85lr7\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.240966 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241028 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241082 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241149 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241261 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241372 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241389 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr8b2\" (UniqueName: \"kubernetes.io/projected/56095386-d7f2-4ad9-a2cf-382a264b67fa-kube-api-access-lr8b2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241408 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56095386-d7f2-4ad9-a2cf-382a264b67fa-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241391 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241494 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.241729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.241625 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.243343 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.243316 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.243829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.243809 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.299295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.299188 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.299295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.299257 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85lr7\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\") pod \"e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.354535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.354313 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:30.774561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.774532 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" Oct 30 19:54:30.775001 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.774575 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx" event={"ID":"56095386-d7f2-4ad9-a2cf-382a264b67fa","Type":"ContainerDied","Data":"d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60"} Oct 30 19:54:30.775001 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.774628 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2f9f2923d45b180d789b14392c4f89570eabf663e7316682e52bcf5abc72f60" Oct 30 19:54:30.786997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.786973 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:54:30.788365 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:54:30.788343 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edb2a17_30c6_4ab1_b51a_d687b7dcaa52.slice/crio-a92ff4f54841128493c4ec9d71d0d5e7ea437e384094a99a70234988e87df930 WatchSource:0}: Error finding container a92ff4f54841128493c4ec9d71d0d5e7ea437e384094a99a70234988e87df930: Status 404 returned error can't find the container with id a92ff4f54841128493c4ec9d71d0d5e7ea437e384094a99a70234988e87df930 Oct 30 19:54:30.791572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.791540 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:30.791685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.791616 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:30.791685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:30.791662 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:31.664830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.664728 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.664830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.664775 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:31.787302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.787270 4789 generic.go:356] "Generic (PLEG): container finished" podID="efac3a37-19a3-4005-93e1-4496391c982d" containerID="7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2" exitCode=0 Oct 30 19:54:31.787759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.787309 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerDied","Data":"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2"} Oct 30 19:54:31.787820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.787783 4789 scope.go:122] "RemoveContainer" containerID="7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2" Oct 30 19:54:31.789774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.789745 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:31.789895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.789835 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:31.789895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.789885 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:31.790016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.789937 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" event={"ID":"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52","Type":"ContainerStarted","Data":"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51"} Oct 30 19:54:31.790016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:31.789979 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" event={"ID":"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52","Type":"ContainerStarted","Data":"a92ff4f54841128493c4ec9d71d0d5e7ea437e384094a99a70234988e87df930"} Oct 30 19:54:32.097238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:32.091599 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podStartSLOduration=3.09157977 podStartE2EDuration="3.09157977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:32.090584451 +0000 UTC m=+4108.592494938" watchObservedRunningTime="2026-10-30 19:54:32.09157977 +0000 UTC m=+4108.593490258" Oct 30 19:54:32.355360 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:32.355328 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:32.356913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:32.356885 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:54:32.357048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:32.356934 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 19:54:32.837232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:32.836571 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerStarted","Data":"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096"} Oct 30 19:54:33.356255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:33.355780 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.356255 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:33.355848 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 19:54:33.430235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:33.429568 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:54:33.430235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:33.429704 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.430235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:33.429754 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:34.355002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:34.354956 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:54:34.355464 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:34.355040 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 19:54:34.430235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:34.429716 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:34.430235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:34.429793 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:35.355615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:35.355584 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:54:35.356166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:35.356138 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 19:54:35.434234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:35.425973 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:35.434234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:35.426025 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:36.356711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:36.356676 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 19:54:36.357369 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:36.357321 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 19:54:36.430234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:36.429559 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:36.430234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:36.429625 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:37.359499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:37.359463 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:37.360000 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:37.359810 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:37.360566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:37.360544 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:54:37.426611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:37.426578 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 19:54:37.426764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:37.426629 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 19:54:38.431730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:38.431700 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:54:38.432247 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:38.432069 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:54:38.432696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:38.432677 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:54:44.787138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:44.787067 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" podUID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" containerName="main" containerID="cri-o://a51ffeef3bf4dc364c3729d18325f8dfdc0327daf0613754dd63648e7be97620" gracePeriod=45 Oct 30 19:54:44.936689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:44.936649 4789 generic.go:356] "Generic (PLEG): container finished" podID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" containerID="a51ffeef3bf4dc364c3729d18325f8dfdc0327daf0613754dd63648e7be97620" exitCode=0 Oct 30 19:54:44.936892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:44.936706 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" event={"ID":"3284beae-ea6d-48b7-ae67-c3fab4c1552c","Type":"ContainerDied","Data":"a51ffeef3bf4dc364c3729d18325f8dfdc0327daf0613754dd63648e7be97620"} Oct 30 19:54:45.081017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.080992 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:54:45.211293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211259 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.211460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211350 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.211460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211399 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.211629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211500 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.211629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211598 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.211838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.211817 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.212042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212013 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.212118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212101 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.212244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212199 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\" (UniqueName: \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\") pod \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\" (UID: \"3284beae-ea6d-48b7-ae67-c3fab4c1552c\") " Oct 30 19:54:45.212706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212684 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.212802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212709 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.212802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.212723 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.213802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.213775 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:45.214304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.214282 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:45.214570 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.214541 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw" pod "3284beae-ea6d-48b7-ae67-c3fab4c1552c" (UID: "3284beae-ea6d-48b7-ae67-c3fab4c1552c"). InnerVolumeSpecName "kube-api-access-fpjkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:45.313422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.313362 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3284beae-ea6d-48b7-ae67-c3fab4c1552c-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.313422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.313385 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3284beae-ea6d-48b7-ae67-c3fab4c1552c-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.313422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.313397 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpjkw\" (UniqueName: \"kubernetes.io/projected/3284beae-ea6d-48b7-ae67-c3fab4c1552c-kube-api-access-fpjkw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.945680 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.945642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" event={"ID":"3284beae-ea6d-48b7-ae67-c3fab4c1552c","Type":"ContainerDied","Data":"f22e37a1701df91d7c55a295a002be12af117f69489047c0c563aaff30c10ca9"} Oct 30 19:54:45.946109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.945659 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v" Oct 30 19:54:45.946109 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:45.945690 4789 scope.go:122] "RemoveContainer" containerID="a51ffeef3bf4dc364c3729d18325f8dfdc0327daf0613754dd63648e7be97620" Oct 30 19:54:46.102354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:46.102322 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:54:46.148204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:46.148172 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b330be/explicit-model-b330be-kserve-847778fc-vll2v"] Oct 30 19:54:47.669055 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:47.669025 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" path="/var/lib/kubelet/pods/3284beae-ea6d-48b7-ae67-c3fab4c1552c/volumes" Oct 30 19:54:48.197691 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:54:48.197650 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:54:57.901019 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:57.900974 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh"] Oct 30 19:54:57.944973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:57.944932 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-17216f-cj8rh"] Oct 30 19:54:58.905711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:58.905676 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2"] Oct 30 19:54:58.951634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:58.951605 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b330be-nv9q2"] Oct 30 19:54:59.670316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:59.670272 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442a3725-c4c5-4b1e-a233-0a24ba92e431" path="/var/lib/kubelet/pods/442a3725-c4c5-4b1e-a233-0a24ba92e431/volumes" Oct 30 19:54:59.670862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:54:59.670843 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8232f2-a08c-487e-9cec-a752da7594d2" path="/var/lib/kubelet/pods/bd8232f2-a08c-487e-9cec-a752da7594d2/volumes" Oct 30 19:55:02.048290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:02.048249 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p"] Oct 30 19:55:02.094531 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:02.094488 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9c8d2a6-nkg6p"] Oct 30 19:55:03.669175 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:03.669138 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2725ae76-8380-4677-aed3-7a6486c46319" path="/var/lib/kubelet/pods/2725ae76-8380-4677-aed3-7a6486c46319/volumes" Oct 30 19:55:04.908281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:04.908248 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:55:04.908670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:04.908435 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" containerID="cri-o://396fdf5f274bf46e8cb370ee4567d8a898db755e108d65f1a787842faef2e47e" gracePeriod=30 Oct 30 19:55:05.567698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.567665 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-17216f-openshift-default-74975945-svk2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:05.567874 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.567728 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:05.711801 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.711763 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6"] Oct 30 19:55:05.712669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.712648 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" containerName="main" Oct 30 19:55:05.712731 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.712673 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" containerName="main" Oct 30 19:55:05.712868 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.712857 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3284beae-ea6d-48b7-ae67-c3fab4c1552c" containerName="main" Oct 30 19:55:05.716782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.716760 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.765991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.765962 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:55:05.766233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.766000 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:55:05.769637 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.769611 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6"] Oct 30 19:55:05.771125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.771100 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.771319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.771295 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw627\" (UniqueName: \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.771588 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.771565 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.781149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.781125 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:05.811363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.811339 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:55:05.872709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.872643 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw627\" (UniqueName: \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.872854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.872834 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.872948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.872878 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.873604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.873584 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.875161 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.875143 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:05.930048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:05.930024 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw627\" (UniqueName: \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:06.031524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.031500 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:06.416674 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:06.416647 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03517dc_81df_4287_a1aa_8e17931037b7.slice/crio-74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207 WatchSource:0}: Error finding container 74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207: Status 404 returned error can't find the container with id 74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207 Oct 30 19:55:06.417119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.417092 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6"] Oct 30 19:55:06.924166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.924138 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:55:06.924450 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.924405 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" containerID="cri-o://cd472d212658a1bdb3413597114d0945efe4053f3cbc6a3571cad54a4b7f7670" gracePeriod=30 Oct 30 19:55:06.927340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.927311 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:55:06.927509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:06.927488 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" containerID="cri-o://1e9924635cecef3700c4525e4d77f196703d8753bb278040af84ba21c171de4b" gracePeriod=30 Oct 30 19:55:07.144251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.144138 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" event={"ID":"a03517dc-81df-4287-a1aa-8e17931037b7","Type":"ContainerStarted","Data":"41098850b6197d181109a398222b63abe63823d32fc13f54ade6dbdf3b7bc19f"} Oct 30 19:55:07.144251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.144191 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" event={"ID":"a03517dc-81df-4287-a1aa-8e17931037b7","Type":"ContainerStarted","Data":"74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207"} Oct 30 19:55:07.257451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.257418 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:07.257609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.257468 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:07.264339 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.264308 4789 patch_prober.go:29] interesting pod/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:07.264448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.264359 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:07.276292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.276247 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" podStartSLOduration=2.276203941 podStartE2EDuration="2.276203941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:07.274702969 +0000 UTC m=+4143.776613456" watchObservedRunningTime="2026-10-30 19:55:07.276203941 +0000 UTC m=+4143.778114452" Oct 30 19:55:07.400997 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.400928 4789 scope.go:122] "RemoveContainer" containerID="e134bf5e08de0f5f5aa0dcbdc637a23bfb3e9d40d8767df6179e7ccd912bb7e6" Oct 30 19:55:07.410606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:07.410588 4789 scope.go:122] "RemoveContainer" containerID="7135e4f79abecda9e7ebcae12b0b1272cd5a7988f42fc5f05d7220d8d3b77d55" Oct 30 19:55:08.152024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:08.151988 4789 generic.go:356] "Generic (PLEG): container finished" podID="a03517dc-81df-4287-a1aa-8e17931037b7" containerID="41098850b6197d181109a398222b63abe63823d32fc13f54ade6dbdf3b7bc19f" exitCode=0 Oct 30 19:55:08.152481 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:08.152074 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" event={"ID":"a03517dc-81df-4287-a1aa-8e17931037b7","Type":"ContainerDied","Data":"41098850b6197d181109a398222b63abe63823d32fc13f54ade6dbdf3b7bc19f"} Oct 30 19:55:09.333157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.333123 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:09.412407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.412332 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\" (UniqueName: \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\") pod \"a03517dc-81df-4287-a1aa-8e17931037b7\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " Oct 30 19:55:09.412614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.412497 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\") pod \"a03517dc-81df-4287-a1aa-8e17931037b7\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " Oct 30 19:55:09.412614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.412519 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\") pod \"a03517dc-81df-4287-a1aa-8e17931037b7\" (UID: \"a03517dc-81df-4287-a1aa-8e17931037b7\") " Oct 30 19:55:09.412916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.412883 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca" pod "a03517dc-81df-4287-a1aa-8e17931037b7" (UID: "a03517dc-81df-4287-a1aa-8e17931037b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:09.414722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.414681 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627" pod "a03517dc-81df-4287-a1aa-8e17931037b7" (UID: "a03517dc-81df-4287-a1aa-8e17931037b7"). InnerVolumeSpecName "kube-api-access-kw627". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:09.414951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.414922 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca" pod "a03517dc-81df-4287-a1aa-8e17931037b7" (UID: "a03517dc-81df-4287-a1aa-8e17931037b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:09.433700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.433673 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 19:55:09.434625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.434604 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a03517dc-81df-4287-a1aa-8e17931037b7" containerName="revoke-keys" Oct 30 19:55:09.434738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.434628 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03517dc-81df-4287-a1aa-8e17931037b7" containerName="revoke-keys" Oct 30 19:55:09.434842 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.434829 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03517dc-81df-4287-a1aa-8e17931037b7" containerName="revoke-keys" Oct 30 19:55:09.440926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.440908 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.479779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.479746 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 19:55:09.492142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.489976 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d50b07-openshift-default-dockercfg-mw7hr\"" Oct 30 19:55:09.514482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.514453 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c44kv\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.514767 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.514745 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.514938 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.514920 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515119 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515102 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515243 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515399 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515523 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515556 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515652 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.515814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515797 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.515886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515821 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03517dc-81df-4287-a1aa-8e17931037b7-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.515886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.515837 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw627\" (UniqueName: \"kubernetes.io/projected/a03517dc-81df-4287-a1aa-8e17931037b7-kube-api-access-kw627\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.617826 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.617890 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.617930 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.617966 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.618080 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.618187 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.618235 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.618281 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.618768 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.618324 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c44kv\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.619742 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.620404 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.620654 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.620887 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.621131 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.630232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.623855 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.633764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.633745 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.676694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.676636 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.679896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.679864 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c44kv\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\") pod \"e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:09.772475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:09.772441 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:10.172039 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.172002 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" Oct 30 19:55:10.172918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.172886 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6" event={"ID":"a03517dc-81df-4287-a1aa-8e17931037b7","Type":"ContainerDied","Data":"74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207"} Oct 30 19:55:10.174284 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.173808 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a8e99a880f541c324dce1000d2aab84e559ec8a1affd952dbc316740bf8207" Oct 30 19:55:10.177677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.177646 4789 generic.go:356] "Generic (PLEG): container finished" podID="98209702-5b61-426c-9470-168fa30d8f04" containerID="396fdf5f274bf46e8cb370ee4567d8a898db755e108d65f1a787842faef2e47e" exitCode=0 Oct 30 19:55:10.177810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.177722 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerDied","Data":"396fdf5f274bf46e8cb370ee4567d8a898db755e108d65f1a787842faef2e47e"} Oct 30 19:55:10.177810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.177792 4789 scope.go:122] "RemoveContainer" containerID="6c160a6f7002f2d9c9393efae0ddef1a2000bc5ed306f7535b054d84056a09ab" Oct 30 19:55:10.189461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.189435 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:55:10.209259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.209230 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 19:55:10.211490 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:10.211454 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551c106f_e5b9_47c2_a48d_78eec02a7561.slice/crio-23f78ad36fafbbef6c9984782d2f61d2f90ccae43b0b4c9d409d37acd6fe7003 WatchSource:0}: Error finding container 23f78ad36fafbbef6c9984782d2f61d2f90ccae43b0b4c9d409d37acd6fe7003: Status 404 returned error can't find the container with id 23f78ad36fafbbef6c9984782d2f61d2f90ccae43b0b4c9d409d37acd6fe7003 Oct 30 19:55:10.214831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.214795 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:10.214947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.214896 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:10.215014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.214942 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:10.333442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333411 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333489 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333562 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333660 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333713 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333783 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333785 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:10.333847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333845 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.334271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333950 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.334271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333956 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:10.334271 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.333992 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\") pod \"98209702-5b61-426c-9470-168fa30d8f04\" (UID: \"98209702-5b61-426c-9470-168fa30d8f04\") " Oct 30 19:55:10.334569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.334545 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98209702-5b61-426c-9470-168fa30d8f04-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.334670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.334574 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.334773 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.334751 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:10.335100 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.335074 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:10.335437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.335415 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:10.336678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.336651 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:10.337167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.337140 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:10.337776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.337752 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:10.337776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.337762 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k" pod "98209702-5b61-426c-9470-168fa30d8f04" (UID: "98209702-5b61-426c-9470-168fa30d8f04"). InnerVolumeSpecName "kube-api-access-qqj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:10.435483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435440 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqj6k\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-kube-api-access-qqj6k\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435477 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435483 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435486 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98209702-5b61-426c-9470-168fa30d8f04-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435495 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435503 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98209702-5b61-426c-9470-168fa30d8f04-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435513 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.435789 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:10.435524 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98209702-5b61-426c-9470-168fa30d8f04-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:11.186746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.186710 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerStarted","Data":"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7"} Oct 30 19:55:11.186934 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.186767 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerStarted","Data":"23f78ad36fafbbef6c9984782d2f61d2f90ccae43b0b4c9d409d37acd6fe7003"} Oct 30 19:55:11.188251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.188194 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" event={"ID":"98209702-5b61-426c-9470-168fa30d8f04","Type":"ContainerDied","Data":"71b7fe6d111c352c718e6e25a9caab933b2ce69723a297a0ad0206e26d44c52d"} Oct 30 19:55:11.188399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.188279 4789 scope.go:122] "RemoveContainer" containerID="396fdf5f274bf46e8cb370ee4567d8a898db755e108d65f1a787842faef2e47e" Oct 30 19:55:11.188399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.188232 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t" Oct 30 19:55:11.350351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.350303 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podStartSLOduration=2.350288589 podStartE2EDuration="2.350288589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.348591374 +0000 UTC m=+4147.850501871" watchObservedRunningTime="2026-10-30 19:55:11.350288589 +0000 UTC m=+4147.852199075" Oct 30 19:55:11.520685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.520606 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:55:11.525171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.525146 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-17216f-openshift-default-74975945-svk2t"] Oct 30 19:55:11.671392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.671355 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98209702-5b61-426c-9470-168fa30d8f04" path="/var/lib/kubelet/pods/98209702-5b61-426c-9470-168fa30d8f04/volumes" Oct 30 19:55:11.772818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.772755 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:11.778677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:11.778654 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:12.050327 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.050303 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd"] Oct 30 19:55:12.054861 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.054832 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9749efc7-9vqsd"] Oct 30 19:55:12.215803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.215773 4789 generic.go:356] "Generic (PLEG): container finished" podID="d2a422e4-d688-468b-b210-44aa1f347e42" containerID="cd472d212658a1bdb3413597114d0945efe4053f3cbc6a3571cad54a4b7f7670" exitCode=0 Oct 30 19:55:12.216047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.216025 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" event={"ID":"d2a422e4-d688-468b-b210-44aa1f347e42","Type":"ContainerDied","Data":"cd472d212658a1bdb3413597114d0945efe4053f3cbc6a3571cad54a4b7f7670"} Oct 30 19:55:12.219557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.219532 4789 generic.go:356] "Generic (PLEG): container finished" podID="42d49601-86c3-41d4-a922-68f4b660d039" containerID="1e9924635cecef3700c4525e4d77f196703d8753bb278040af84ba21c171de4b" exitCode=0 Oct 30 19:55:12.220785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.220763 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" event={"ID":"42d49601-86c3-41d4-a922-68f4b660d039","Type":"ContainerDied","Data":"1e9924635cecef3700c4525e4d77f196703d8753bb278040af84ba21c171de4b"} Oct 30 19:55:12.220946 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.220932 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:12.222108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.222090 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:55:12.244548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.244527 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:55:12.253280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.253257 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:55:12.355062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.354959 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355045 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355078 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355145 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355192 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355268 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355317 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355359 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355401 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355439 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355487 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355558 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355596 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.355627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355588 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355640 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355652 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355667 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355722 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355765 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355795 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\") pod \"42d49601-86c3-41d4-a922-68f4b660d039\" (UID: \"42d49601-86c3-41d4-a922-68f4b660d039\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355827 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\") pod \"d2a422e4-d688-468b-b210-44aa1f347e42\" (UID: \"d2a422e4-d688-468b-b210-44aa1f347e42\") " Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355836 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.355885 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356010 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356350 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356399 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a422e4-d688-468b-b210-44aa1f347e42-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356420 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356434 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356450 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356485 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356512 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.356808 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.357625 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.357601 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.358741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.357758 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.358741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.358020 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:12.359630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.359589 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:12.359863 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.359840 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:12.360235 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.360186 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.360687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.360663 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:12.361138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.361120 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:12.361299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.361155 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd" pod "d2a422e4-d688-468b-b210-44aa1f347e42" (UID: "d2a422e4-d688-468b-b210-44aa1f347e42"). InnerVolumeSpecName "kube-api-access-5h8cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:12.361498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.361474 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw" pod "42d49601-86c3-41d4-a922-68f4b660d039" (UID: "42d49601-86c3-41d4-a922-68f4b660d039"). InnerVolumeSpecName "kube-api-access-5gpdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:12.458044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458008 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a422e4-d688-468b-b210-44aa1f347e42-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458045 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458064 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h8cd\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-kube-api-access-5h8cd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458080 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a422e4-d688-468b-b210-44aa1f347e42-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458094 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458108 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458122 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpdw\" (UniqueName: \"kubernetes.io/projected/42d49601-86c3-41d4-a922-68f4b660d039-kube-api-access-5gpdw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458137 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d49601-86c3-41d4-a922-68f4b660d039-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458151 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a422e4-d688-468b-b210-44aa1f347e42-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458165 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d49601-86c3-41d4-a922-68f4b660d039-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458178 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:12.458265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:12.458191 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d49601-86c3-41d4-a922-68f4b660d039-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.230379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.230289 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" Oct 30 19:55:13.230560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.230282 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k" event={"ID":"d2a422e4-d688-468b-b210-44aa1f347e42","Type":"ContainerDied","Data":"0d18bcc0d6d7fa5a068299526f83ed1b0eb637cc1416f162bca7f30a37db3064"} Oct 30 19:55:13.230560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.230439 4789 scope.go:122] "RemoveContainer" containerID="cd472d212658a1bdb3413597114d0945efe4053f3cbc6a3571cad54a4b7f7670" Oct 30 19:55:13.231995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.231965 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" Oct 30 19:55:13.232099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.232005 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh" event={"ID":"42d49601-86c3-41d4-a922-68f4b660d039","Type":"ContainerDied","Data":"832de4f771aac3d5f5b4257589da7c90b45d2a43179cf38c4a95dbe8efab58d0"} Oct 30 19:55:13.243761 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.243740 4789 scope.go:122] "RemoveContainer" containerID="1e9924635cecef3700c4525e4d77f196703d8753bb278040af84ba21c171de4b" Oct 30 19:55:13.387282 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.387251 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:55:13.411839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.411807 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-k6x6k"] Oct 30 19:55:13.496654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.496576 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:55:13.500901 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.500873 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b330be-openshift-default-7d97dd6497-wndjh"] Oct 30 19:55:13.669163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.669133 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d49601-86c3-41d4-a922-68f4b660d039" path="/var/lib/kubelet/pods/42d49601-86c3-41d4-a922-68f4b660d039/volumes" Oct 30 19:55:13.669615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.669595 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba6458b-ead0-4ac4-b6ba-bc1559b10b43" path="/var/lib/kubelet/pods/5ba6458b-ead0-4ac4-b6ba-bc1559b10b43/volumes" Oct 30 19:55:13.670025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:13.670012 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" path="/var/lib/kubelet/pods/d2a422e4-d688-468b-b210-44aa1f347e42/volumes" Oct 30 19:55:17.818806 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.818775 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 19:55:17.819495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819474 4789 cpu_manager.go:414] "containerMap: removing container" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" Oct 30 19:55:17.819495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819494 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819518 4789 cpu_manager.go:414] "containerMap: removing container" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819525 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819548 4789 cpu_manager.go:414] "containerMap: removing container" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819556 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819565 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" Oct 30 19:55:17.819655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819570 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" Oct 30 19:55:17.819971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819684 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:17.819971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819691 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a422e4-d688-468b-b210-44aa1f347e42" containerName="istio-proxy" Oct 30 19:55:17.819971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.819705 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d49601-86c3-41d4-a922-68f4b660d039" containerName="istio-proxy" Oct 30 19:55:17.824892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.824867 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:17.874412 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.874387 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d50b07\"" Oct 30 19:55:17.924375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.924336 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:17.924561 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.924454 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62b2s\" (UniqueName: \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:17.924986 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.924961 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d50b07-dockercfg-m5vmg\"" Oct 30 19:55:17.928138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:17.928112 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 19:55:18.000126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.000092 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 19:55:18.000829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.000810 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="98209702-5b61-426c-9470-168fa30d8f04" containerName="istio-proxy" Oct 30 19:55:18.004651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.004628 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.026078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.026051 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:18.026217 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.026153 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62b2s\" (UniqueName: \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:18.026811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.026792 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:18.059008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.058983 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 19:55:18.084557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.084509 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62b2s\" (UniqueName: \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\") pod \"payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:18.127608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.127578 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.127754 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.127738 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq274\" (UniqueName: \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.141221 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.141192 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:18.220474 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:18.220426 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:55:18.229333 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.229300 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq274\" (UniqueName: \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.229502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.229480 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.229973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.229878 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg"] Oct 30 19:55:18.230512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.230435 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.238486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.236075 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.285566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.285538 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:55:18.285704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.285659 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:55:18.285772 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.285735 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:18.286230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.286187 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:55:18.288615 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.288586 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq274\" (UniqueName: \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\") pod \"payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.289423 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.289404 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg"] Oct 30 19:55:18.319259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.319196 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:18.330979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.330951 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.331086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.331005 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.331086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.331044 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4rv\" (UniqueName: \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.431763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.431730 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.431989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.431788 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.431989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.431848 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4rv\" (UniqueName: \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.432584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.432557 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.434244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.434199 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.489573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.489552 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4rv\" (UniqueName: \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\") pod \"maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.525600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.525575 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 19:55:18.528017 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:18.527992 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b5c23e_4020_4358_8b01_183077b68965.slice/crio-b3ddccc821225b09c3945b0d0a34a045fb81c2538d2dd9004148b8bc602adbbf WatchSource:0}: Error finding container b3ddccc821225b09c3945b0d0a34a045fb81c2538d2dd9004148b8bc602adbbf: Status 404 returned error can't find the container with id b3ddccc821225b09c3945b0d0a34a045fb81c2538d2dd9004148b8bc602adbbf Oct 30 19:55:18.553702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.553665 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:18.691959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.691929 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 19:55:18.697062 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:18.697030 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc44738_c830_4793_b8f9_2ab533a84f4e.slice/crio-7ebe4a17d5033ec87ab4f8041307de86e423f5a5ff8a5550efbedf2188865f35 WatchSource:0}: Error finding container 7ebe4a17d5033ec87ab4f8041307de86e423f5a5ff8a5550efbedf2188865f35: Status 404 returned error can't find the container with id 7ebe4a17d5033ec87ab4f8041307de86e423f5a5ff8a5550efbedf2188865f35 Oct 30 19:55:18.937172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:18.937145 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg"] Oct 30 19:55:18.937608 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:18.937588 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf853fe2_150f_4ddd_a759_90fd7a6d9b5b.slice/crio-fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae WatchSource:0}: Error finding container fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae: Status 404 returned error can't find the container with id fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae Oct 30 19:55:19.282654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.282617 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" event={"ID":"f6b5c23e-4020-4358-8b01-183077b68965","Type":"ContainerStarted","Data":"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe"} Oct 30 19:55:19.282831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.282706 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:19.282831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.282732 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" event={"ID":"f6b5c23e-4020-4358-8b01-183077b68965","Type":"ContainerStarted","Data":"b3ddccc821225b09c3945b0d0a34a045fb81c2538d2dd9004148b8bc602adbbf"} Oct 30 19:55:19.284445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.284407 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" event={"ID":"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b","Type":"ContainerStarted","Data":"ec4fb5a165b49bd7f1bb21aa4ade2cfa69c13c744f3f2b42ca224fdce1e1a0a9"} Oct 30 19:55:19.284596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.284454 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" event={"ID":"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b","Type":"ContainerStarted","Data":"fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae"} Oct 30 19:55:19.286077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.286051 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" event={"ID":"3cc44738-c830-4793-b8f9-2ab533a84f4e","Type":"ContainerStarted","Data":"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6"} Oct 30 19:55:19.286232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.286097 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" event={"ID":"3cc44738-c830-4793-b8f9-2ab533a84f4e","Type":"ContainerStarted","Data":"7ebe4a17d5033ec87ab4f8041307de86e423f5a5ff8a5550efbedf2188865f35"} Oct 30 19:55:19.286232 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.286127 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:19.393856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.393791 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" podStartSLOduration=2.393774639 podStartE2EDuration="2.393774639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:19.391967131 +0000 UTC m=+4155.893877631" watchObservedRunningTime="2026-10-30 19:55:19.393774639 +0000 UTC m=+4155.895685126" Oct 30 19:55:19.501838 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.501649 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" podStartSLOduration=1.501631407 podStartE2EDuration="1.501631407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:19.501086402 +0000 UTC m=+4156.002996884" watchObservedRunningTime="2026-10-30 19:55:19.501631407 +0000 UTC m=+4156.003541894" Oct 30 19:55:19.613911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.613858 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" podStartSLOduration=2.613845005 podStartE2EDuration="2.613845005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:19.610166579 +0000 UTC m=+4156.112077101" watchObservedRunningTime="2026-10-30 19:55:19.613845005 +0000 UTC m=+4156.115755491" Oct 30 19:55:19.828543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:19.828461 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:55:20.296166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:20.296124 4789 generic.go:356] "Generic (PLEG): container finished" podID="bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" containerID="ec4fb5a165b49bd7f1bb21aa4ade2cfa69c13c744f3f2b42ca224fdce1e1a0a9" exitCode=0 Oct 30 19:55:20.296678 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:20.296220 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" event={"ID":"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b","Type":"ContainerDied","Data":"ec4fb5a165b49bd7f1bb21aa4ade2cfa69c13c744f3f2b42ca224fdce1e1a0a9"} Oct 30 19:55:21.480780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.480750 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:21.676148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.676081 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\" (UniqueName: \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\") pod \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " Oct 30 19:55:21.676148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.676145 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\") pod \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " Oct 30 19:55:21.676345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.676250 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\") pod \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\" (UID: \"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b\") " Oct 30 19:55:21.676581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.676559 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca" pod "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" (UID: "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:21.678250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.678198 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv" pod "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" (UID: "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b"). InnerVolumeSpecName "kube-api-access-dc4rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:21.678357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.678246 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca" pod "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" (UID: "bf853fe2-150f-4ddd-a759-90fd7a6d9b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:21.778136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.778113 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4rv\" (UniqueName: \"kubernetes.io/projected/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-kube-api-access-dc4rv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:21.778307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.778151 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:21.778307 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:21.778168 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:22.311136 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:22.311105 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" Oct 30 19:55:22.311303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:22.311139 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg" event={"ID":"bf853fe2-150f-4ddd-a759-90fd7a6d9b5b","Type":"ContainerDied","Data":"fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae"} Oct 30 19:55:22.311303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:22.311188 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd53f07c0cbd1d447a680f7edfdbc0fe327608cfa92debd93f0c075eec030aae" Oct 30 19:55:24.829239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:24.829188 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" podUID="3b974bca-d824-4838-a08b-5f8304fc16c9" containerName="payload-pre-processing" containerID="cri-o://d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7" gracePeriod=25 Oct 30 19:55:25.093630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.093573 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:55:25.117766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.117732 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\" (UniqueName: \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\") pod \"3b974bca-d824-4838-a08b-5f8304fc16c9\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " Oct 30 19:55:25.117904 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.117887 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\") pod \"3b974bca-d824-4838-a08b-5f8304fc16c9\" (UID: \"3b974bca-d824-4838-a08b-5f8304fc16c9\") " Oct 30 19:55:25.118299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.118268 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume" pod "3b974bca-d824-4838-a08b-5f8304fc16c9" (UID: "3b974bca-d824-4838-a08b-5f8304fc16c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:25.118442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.118421 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b974bca-d824-4838-a08b-5f8304fc16c9-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:25.120042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.120020 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2" pod "3b974bca-d824-4838-a08b-5f8304fc16c9" (UID: "3b974bca-d824-4838-a08b-5f8304fc16c9"). InnerVolumeSpecName "kube-api-access-dd4j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:25.219407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.219377 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4j2\" (UniqueName: \"kubernetes.io/projected/3b974bca-d824-4838-a08b-5f8304fc16c9-kube-api-access-dd4j2\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:25.332262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.332231 4789 generic.go:356] "Generic (PLEG): container finished" podID="3b974bca-d824-4838-a08b-5f8304fc16c9" containerID="d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7" exitCode=0 Oct 30 19:55:25.332371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.332280 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" Oct 30 19:55:25.332371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.332326 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" event={"ID":"3b974bca-d824-4838-a08b-5f8304fc16c9","Type":"ContainerDied","Data":"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7"} Oct 30 19:55:25.332503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.332384 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d" event={"ID":"3b974bca-d824-4838-a08b-5f8304fc16c9","Type":"ContainerDied","Data":"d8ba5efc5d90afe36f6d3822f519bb4a215a57d8fbbdcc131605566810ae297e"} Oct 30 19:55:25.332503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.332417 4789 scope.go:122] "RemoveContainer" containerID="d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7" Oct 30 19:55:25.343371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.343352 4789 scope.go:122] "RemoveContainer" containerID="d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7" Oct 30 19:55:25.343653 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:25.343608 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7\": container with ID starting with d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7 not found: ID does not exist" containerID="d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7" Oct 30 19:55:25.343653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.343637 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7"} err="failed to get container status \"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7\": rpc error: code = NotFound desc = could not find container \"d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7\": container with ID starting with d92d0fde0297ab85a51e6c6c862c48101a11deb80cbaf884178d4c6a26d382a7 not found: ID does not exist" Oct 30 19:55:25.496790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.496763 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:55:25.499642 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.499618 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4a36b7-8685ccc667-l5m4d"] Oct 30 19:55:25.670266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:25.670171 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b974bca-d824-4838-a08b-5f8304fc16c9" path="/var/lib/kubelet/pods/3b974bca-d824-4838-a08b-5f8304fc16c9/volumes" Oct 30 19:55:28.634023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.633985 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp"] Oct 30 19:55:28.634856 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.634832 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3b974bca-d824-4838-a08b-5f8304fc16c9" containerName="payload-pre-processing" Oct 30 19:55:28.634908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.634860 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b974bca-d824-4838-a08b-5f8304fc16c9" containerName="payload-pre-processing" Oct 30 19:55:28.634908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.634887 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" containerName="revoke-keys" Oct 30 19:55:28.634908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.634894 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" containerName="revoke-keys" Oct 30 19:55:28.635112 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.635099 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b974bca-d824-4838-a08b-5f8304fc16c9" containerName="payload-pre-processing" Oct 30 19:55:28.635147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.635118 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" containerName="revoke-keys" Oct 30 19:55:28.640179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.640156 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.692742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.692709 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp"] Oct 30 19:55:28.734574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.734544 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:55:28.734739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.734636 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:55:28.734739 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.734685 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:55:28.735144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.735123 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:28.762931 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.762905 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.763120 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.763089 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmq8f\" (UniqueName: \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.763279 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.763259 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.864092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.864060 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.864268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.864147 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmq8f\" (UniqueName: \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.864268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.864200 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.864689 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.864667 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.866519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.866495 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.922023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.921966 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmq8f\" (UniqueName: \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:28.953915 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:28.953897 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:29.335424 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:29.335393 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp"] Oct 30 19:55:29.337238 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:29.337182 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ed85ae_43a6_43f0_8f8a_d80053838d8a.slice/crio-db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb WatchSource:0}: Error finding container db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb: Status 404 returned error can't find the container with id db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb Oct 30 19:55:29.363080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:29.363054 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" event={"ID":"20ed85ae-43a6-43f0-8f8a-d80053838d8a","Type":"ContainerStarted","Data":"db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb"} Oct 30 19:55:30.299841 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.299806 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 19:55:30.300251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.300020 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 19:55:30.371792 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.371761 4789 generic.go:356] "Generic (PLEG): container finished" podID="20ed85ae-43a6-43f0-8f8a-d80053838d8a" containerID="376a1c4184afdd9a60ec2ddf5ab977e239e66e123052c7dd6a80de764a787bb8" exitCode=0 Oct 30 19:55:30.371963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.371820 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" event={"ID":"20ed85ae-43a6-43f0-8f8a-d80053838d8a","Type":"ContainerDied","Data":"376a1c4184afdd9a60ec2ddf5ab977e239e66e123052c7dd6a80de764a787bb8"} Oct 30 19:55:30.448074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.448041 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:55:30.549289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:30.549254 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:55:31.546597 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.546570 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:31.593787 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.593754 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\") pod \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " Oct 30 19:55:31.593943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.593811 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\") pod \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " Oct 30 19:55:31.593943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.593875 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\" (UniqueName: \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\") pod \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\" (UID: \"20ed85ae-43a6-43f0-8f8a-d80053838d8a\") " Oct 30 19:55:31.594239 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.594197 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca" pod "20ed85ae-43a6-43f0-8f8a-d80053838d8a" (UID: "20ed85ae-43a6-43f0-8f8a-d80053838d8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:31.594547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.594514 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:31.596497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.596469 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f" pod "20ed85ae-43a6-43f0-8f8a-d80053838d8a" (UID: "20ed85ae-43a6-43f0-8f8a-d80053838d8a"). InnerVolumeSpecName "kube-api-access-hmq8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:31.596595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.596494 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca" pod "20ed85ae-43a6-43f0-8f8a-d80053838d8a" (UID: "20ed85ae-43a6-43f0-8f8a-d80053838d8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:31.695393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.695363 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ed85ae-43a6-43f0-8f8a-d80053838d8a-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:31.695393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:31.695393 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmq8f\" (UniqueName: \"kubernetes.io/projected/20ed85ae-43a6-43f0-8f8a-d80053838d8a-kube-api-access-hmq8f\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:32.386364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:32.386334 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" Oct 30 19:55:32.386364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:32.386345 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp" event={"ID":"20ed85ae-43a6-43f0-8f8a-d80053838d8a","Type":"ContainerDied","Data":"db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb"} Oct 30 19:55:32.386587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:32.386409 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6d54006d8f915b389e01eefde4897a2d79eda0ddca9e286d5007b08e3082cb" Oct 30 19:55:35.448396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.448360 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" podUID="26cd257a-812f-416f-842c-a6f67fbf290d" containerName="payload-processing" containerID="cri-o://9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b" gracePeriod=25 Oct 30 19:55:35.549670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.549635 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerName="payload-pre-processing" containerID="cri-o://ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017" gracePeriod=25 Oct 30 19:55:35.741867 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.741836 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:55:35.807034 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.807011 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:55:35.841325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.841285 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\" (UniqueName: \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\") pod \"26cd257a-812f-416f-842c-a6f67fbf290d\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " Oct 30 19:55:35.841460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.841429 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\") pod \"26cd257a-812f-416f-842c-a6f67fbf290d\" (UID: \"26cd257a-812f-416f-842c-a6f67fbf290d\") " Oct 30 19:55:35.841779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.841744 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume" pod "26cd257a-812f-416f-842c-a6f67fbf290d" (UID: "26cd257a-812f-416f-842c-a6f67fbf290d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:35.841883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.841857 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26cd257a-812f-416f-842c-a6f67fbf290d-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:35.843351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.843332 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr" pod "26cd257a-812f-416f-842c-a6f67fbf290d" (UID: "26cd257a-812f-416f-842c-a6f67fbf290d"). InnerVolumeSpecName "kube-api-access-pghjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:35.942490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.942462 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\" (UniqueName: \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\") pod \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " Oct 30 19:55:35.942618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.942604 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\") pod \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\" (UID: \"afe7d04c-da06-4e27-9630-d0eb2c0c0769\") " Oct 30 19:55:35.942862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.942849 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghjr\" (UniqueName: \"kubernetes.io/projected/26cd257a-812f-416f-842c-a6f67fbf290d-kube-api-access-pghjr\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:35.942967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.942949 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume" pod "afe7d04c-da06-4e27-9630-d0eb2c0c0769" (UID: "afe7d04c-da06-4e27-9630-d0eb2c0c0769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:35.944457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:35.944438 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m" pod "afe7d04c-da06-4e27-9630-d0eb2c0c0769" (UID: "afe7d04c-da06-4e27-9630-d0eb2c0c0769"). InnerVolumeSpecName "kube-api-access-t2h9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:36.044205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.044134 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe7d04c-da06-4e27-9630-d0eb2c0c0769-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:36.044205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.044162 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2h9m\" (UniqueName: \"kubernetes.io/projected/afe7d04c-da06-4e27-9630-d0eb2c0c0769-kube-api-access-t2h9m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:36.416041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.415965 4789 generic.go:356] "Generic (PLEG): container finished" podID="26cd257a-812f-416f-842c-a6f67fbf290d" containerID="9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b" exitCode=0 Oct 30 19:55:36.416041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.416010 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" Oct 30 19:55:36.416273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.416049 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" event={"ID":"26cd257a-812f-416f-842c-a6f67fbf290d","Type":"ContainerDied","Data":"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b"} Oct 30 19:55:36.416273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.416097 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd" event={"ID":"26cd257a-812f-416f-842c-a6f67fbf290d","Type":"ContainerDied","Data":"5160ba439c75d690ef20c2ab069befe285db5d872954ccfbf2c0a80ce41ee34d"} Oct 30 19:55:36.416273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.416119 4789 scope.go:122] "RemoveContainer" containerID="9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b" Oct 30 19:55:36.417547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.417517 4789 generic.go:356] "Generic (PLEG): container finished" podID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerID="ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017" exitCode=0 Oct 30 19:55:36.417675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.417549 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" event={"ID":"afe7d04c-da06-4e27-9630-d0eb2c0c0769","Type":"ContainerDied","Data":"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017"} Oct 30 19:55:36.417675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.417559 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" Oct 30 19:55:36.417675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.417599 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" event={"ID":"afe7d04c-da06-4e27-9630-d0eb2c0c0769","Type":"ContainerDied","Data":"f0b5fe307e96d112a8d51e0ca143faa9ee4678a3e65add8a9dbbd21ab7f45294"} Oct 30 19:55:36.429944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.429925 4789 scope.go:122] "RemoveContainer" containerID="9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b" Oct 30 19:55:36.430278 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:36.430253 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b\": container with ID starting with 9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b not found: ID does not exist" containerID="9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b" Oct 30 19:55:36.430374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.430288 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b"} err="failed to get container status \"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b\": rpc error: code = NotFound desc = could not find container \"9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b\": container with ID starting with 9edf5b3d0970f72ebc93370ec812863bdbb485f72630a6aa73d1bcece8a3a57b not found: ID does not exist" Oct 30 19:55:36.430374 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.430311 4789 scope.go:122] "RemoveContainer" containerID="ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017" Oct 30 19:55:36.440351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.440320 4789 scope.go:122] "RemoveContainer" containerID="ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017" Oct 30 19:55:36.440640 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:36.440618 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017\": container with ID starting with ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017 not found: ID does not exist" containerID="ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017" Oct 30 19:55:36.440707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.440650 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017"} err="failed to get container status \"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017\": rpc error: code = NotFound desc = could not find container \"ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017\": container with ID starting with ce9db7f4ef5458415c485c2a60f746fa5c32003a68618cd514ef695877385017 not found: ID does not exist" Oct 30 19:55:36.572106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.572078 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:55:36.618302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.618280 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-7b6aee-6c9fdc5f68-wbtdd"] Oct 30 19:55:36.721378 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.721352 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:55:36.724971 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.724950 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4"] Oct 30 19:55:36.778979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:36.778944 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-7b6aee-b99f9c96c-p2nb4" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.34:50052\" within 1s: context deadline exceeded" Oct 30 19:55:36.779116 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:55:36.779009 4789 logging.go:55] [core] [Channel #6135 SubChannel #6136]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.34:50052", ServerName: "10.132.1.34:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.34:50052: operation was canceled" Oct 30 19:55:37.669487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:37.669452 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26cd257a-812f-416f-842c-a6f67fbf290d" path="/var/lib/kubelet/pods/26cd257a-812f-416f-842c-a6f67fbf290d/volumes" Oct 30 19:55:37.669979 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:37.669864 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" path="/var/lib/kubelet/pods/afe7d04c-da06-4e27-9630-d0eb2c0c0769/volumes" Oct 30 19:55:40.022742 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:40.022708 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:55:40.122021 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:40.121990 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:55:45.023781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.023698 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" podUID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" containerName="payload-processing" containerID="cri-o://3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0" gracePeriod=25 Oct 30 19:55:45.123164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.123129 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" podUID="249b213b-d43c-475a-9afa-c5cef977a329" containerName="payload-pre-processing" containerID="cri-o://b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8" gracePeriod=25 Oct 30 19:55:45.290942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.290866 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:55:45.410594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.410567 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:55:45.442478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.442444 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\") pod \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " Oct 30 19:55:45.443078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.442714 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\" (UniqueName: \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\") pod \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\" (UID: \"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a\") " Oct 30 19:55:45.443078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.442828 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume" pod "4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" (UID: "4e238f0b-aac5-4e07-a5c2-46cabac5eb4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:45.443288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.443143 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:45.451366 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.451338 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz" pod "4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" (UID: "4e238f0b-aac5-4e07-a5c2-46cabac5eb4a"). InnerVolumeSpecName "kube-api-access-scbkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:45.485016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.484979 4789 generic.go:356] "Generic (PLEG): container finished" podID="249b213b-d43c-475a-9afa-c5cef977a329" containerID="b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8" exitCode=0 Oct 30 19:55:45.485168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.485025 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" Oct 30 19:55:45.485168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.485067 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" event={"ID":"249b213b-d43c-475a-9afa-c5cef977a329","Type":"ContainerDied","Data":"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8"} Oct 30 19:55:45.485168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.485117 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb" event={"ID":"249b213b-d43c-475a-9afa-c5cef977a329","Type":"ContainerDied","Data":"93394b35313125f7f3561620825fd1f15f183c622be58ffd308bd589dd2290bd"} Oct 30 19:55:45.485168 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.485145 4789 scope.go:122] "RemoveContainer" containerID="b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8" Oct 30 19:55:45.486545 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.486520 4789 generic.go:356] "Generic (PLEG): container finished" podID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" containerID="3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0" exitCode=0 Oct 30 19:55:45.486685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.486548 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" event={"ID":"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a","Type":"ContainerDied","Data":"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0"} Oct 30 19:55:45.486685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.486570 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" Oct 30 19:55:45.486685 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.486603 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs" event={"ID":"4e238f0b-aac5-4e07-a5c2-46cabac5eb4a","Type":"ContainerDied","Data":"60eb4904e371f0de9ddc1ce39af9a5be1148fb12a5eea8107395dd5cc8fe65e0"} Oct 30 19:55:45.496953 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.496932 4789 scope.go:122] "RemoveContainer" containerID="b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8" Oct 30 19:55:45.497361 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:45.497338 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8\": container with ID starting with b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8 not found: ID does not exist" containerID="b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8" Oct 30 19:55:45.497446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.497369 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8"} err="failed to get container status \"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8\": rpc error: code = NotFound desc = could not find container \"b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8\": container with ID starting with b7b22682e6bb73dad2b30cc78acea834e32689bf79e641f9c730d23f058999b8 not found: ID does not exist" Oct 30 19:55:45.497446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.497385 4789 scope.go:122] "RemoveContainer" containerID="3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0" Oct 30 19:55:45.506337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.506317 4789 scope.go:122] "RemoveContainer" containerID="3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0" Oct 30 19:55:45.506593 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:45.506572 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0\": container with ID starting with 3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0 not found: ID does not exist" containerID="3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0" Oct 30 19:55:45.506658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.506600 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0"} err="failed to get container status \"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0\": rpc error: code = NotFound desc = could not find container \"3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0\": container with ID starting with 3c537332719fdf46ce4010ab41a7a7172a5287bb83a21ba19e7058ce3e4945e0 not found: ID does not exist" Oct 30 19:55:45.543849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.543774 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\" (UniqueName: \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\") pod \"249b213b-d43c-475a-9afa-c5cef977a329\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " Oct 30 19:55:45.543974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.543957 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\") pod \"249b213b-d43c-475a-9afa-c5cef977a329\" (UID: \"249b213b-d43c-475a-9afa-c5cef977a329\") " Oct 30 19:55:45.544388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.544366 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scbkz\" (UniqueName: \"kubernetes.io/projected/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a-kube-api-access-scbkz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:45.544506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.544392 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume" pod "249b213b-d43c-475a-9afa-c5cef977a329" (UID: "249b213b-d43c-475a-9afa-c5cef977a329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:45.545955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.545925 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg" pod "249b213b-d43c-475a-9afa-c5cef977a329" (UID: "249b213b-d43c-475a-9afa-c5cef977a329"). InnerVolumeSpecName "kube-api-access-jt8gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:45.645872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.645836 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/249b213b-d43c-475a-9afa-c5cef977a329-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:45.645872 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.645865 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8gg\" (UniqueName: \"kubernetes.io/projected/249b213b-d43c-475a-9afa-c5cef977a329-kube-api-access-jt8gg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:45.676779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.676754 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:55:45.680733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.680712 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-660b59-7b687c96d4-tqvvs"] Oct 30 19:55:45.896067 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.895997 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:55:45.901940 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:45.901912 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-660b59-6878d789b7-jhlhb"] Oct 30 19:55:47.668525 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:47.668481 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249b213b-d43c-475a-9afa-c5cef977a329" path="/var/lib/kubelet/pods/249b213b-d43c-475a-9afa-c5cef977a329/volumes" Oct 30 19:55:47.668983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:55:47.668956 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" path="/var/lib/kubelet/pods/4e238f0b-aac5-4e07-a5c2-46cabac5eb4a/volumes" Oct 30 19:55:48.196771 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:55:48.196730 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:56:04.624599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:04.624570 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:56:04.654809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:04.654782 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 19:56:07.447722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:07.447685 4789 scope.go:122] "RemoveContainer" containerID="4e7f889c9a965615020cc499f0969661b1397f8259d7cae2945251d60ee2b34d" Oct 30 19:56:07.458010 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:07.457991 4789 scope.go:122] "RemoveContainer" containerID="fea694f46c8323c6f8431126bb61e71d4e8750e668848992515420c3e13790a0" Oct 30 19:56:07.468844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:07.468825 4789 scope.go:122] "RemoveContainer" containerID="45780354ed22032c76f8024f84599f8d154f28ed97ed791d23e41db2a4069b64" Oct 30 19:56:08.278858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:08.278824 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx"] Oct 30 19:56:08.324816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:08.324786 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-905752-ljhtx"] Oct 30 19:56:09.668189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:09.668154 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56095386-d7f2-4ad9-a2cf-382a264b67fa" path="/var/lib/kubelet/pods/56095386-d7f2-4ad9-a2cf-382a264b67fa/volumes" Oct 30 19:56:18.197940 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:56:18.197899 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:56:18.431362 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431327 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p"] Oct 30 19:56:18.431895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431877 4789 cpu_manager.go:414] "containerMap: removing container" podUID="26cd257a-812f-416f-842c-a6f67fbf290d" containerName="payload-processing" Oct 30 19:56:18.431895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431894 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26cd257a-812f-416f-842c-a6f67fbf290d" containerName="payload-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431909 4789 cpu_manager.go:414] "containerMap: removing container" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerName="payload-pre-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431915 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerName="payload-pre-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431925 4789 cpu_manager.go:414] "containerMap: removing container" podUID="20ed85ae-43a6-43f0-8f8a-d80053838d8a" containerName="revoke-keys" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431930 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ed85ae-43a6-43f0-8f8a-d80053838d8a" containerName="revoke-keys" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431943 4789 cpu_manager.go:414] "containerMap: removing container" podUID="249b213b-d43c-475a-9afa-c5cef977a329" containerName="payload-pre-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431948 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249b213b-d43c-475a-9afa-c5cef977a329" containerName="payload-pre-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431978 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" containerName="payload-processing" Oct 30 19:56:18.432047 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.431983 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" containerName="payload-processing" Oct 30 19:56:18.432524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.432071 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="26cd257a-812f-416f-842c-a6f67fbf290d" containerName="payload-processing" Oct 30 19:56:18.432524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.432081 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe7d04c-da06-4e27-9630-d0eb2c0c0769" containerName="payload-pre-processing" Oct 30 19:56:18.432524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.432089 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ed85ae-43a6-43f0-8f8a-d80053838d8a" containerName="revoke-keys" Oct 30 19:56:18.432524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.432096 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="249b213b-d43c-475a-9afa-c5cef977a329" containerName="payload-pre-processing" Oct 30 19:56:18.432524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.432107 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e238f0b-aac5-4e07-a5c2-46cabac5eb4a" containerName="payload-processing" Oct 30 19:56:18.435823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.435801 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.484849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.484791 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:56:18.484849 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.484802 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:18.531123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.531095 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:56:18.531311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.531142 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:56:18.534485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.534460 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p"] Oct 30 19:56:18.597536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.597507 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.597652 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.597627 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpq8n\" (UniqueName: \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.597826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.597804 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.698854 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.698830 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.698974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.698875 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpq8n\" (UniqueName: \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.698974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.698937 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.699602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.699581 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.701252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.701226 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:18.756738 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:18.756688 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpq8n\" (UniqueName: \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:19.049320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:19.049237 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:19.426750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:19.426725 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p"] Oct 30 19:56:19.429775 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:56:19.429744 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a42f50_8414_4c28_b819_b24a76c547f3.slice/crio-7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb WatchSource:0}: Error finding container 7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb: Status 404 returned error can't find the container with id 7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb Oct 30 19:56:19.739569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:19.739536 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" event={"ID":"29a42f50-8414-4c28-b819-b24a76c547f3","Type":"ContainerStarted","Data":"141eb5480a9ad118625ffb7f5100cfbce541c3d7356af435e3fd065b6bdf36a0"} Oct 30 19:56:19.739744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:19.739621 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" event={"ID":"29a42f50-8414-4c28-b819-b24a76c547f3","Type":"ContainerStarted","Data":"7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb"} Oct 30 19:56:19.866690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:19.866638 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" podStartSLOduration=1.866622895 podStartE2EDuration="1.866622895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:19.864033432 +0000 UTC m=+4216.365943931" watchObservedRunningTime="2026-10-30 19:56:19.866622895 +0000 UTC m=+4216.368533405" Oct 30 19:56:20.747296 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:20.747261 4789 generic.go:356] "Generic (PLEG): container finished" podID="29a42f50-8414-4c28-b819-b24a76c547f3" containerID="141eb5480a9ad118625ffb7f5100cfbce541c3d7356af435e3fd065b6bdf36a0" exitCode=0 Oct 30 19:56:20.747732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:20.747348 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" event={"ID":"29a42f50-8414-4c28-b819-b24a76c547f3","Type":"ContainerDied","Data":"141eb5480a9ad118625ffb7f5100cfbce541c3d7356af435e3fd065b6bdf36a0"} Oct 30 19:56:21.895675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:21.895651 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:22.035733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.035646 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\" (UniqueName: \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\") pod \"29a42f50-8414-4c28-b819-b24a76c547f3\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " Oct 30 19:56:22.035911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.035867 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\") pod \"29a42f50-8414-4c28-b819-b24a76c547f3\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " Oct 30 19:56:22.035985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.035944 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\") pod \"29a42f50-8414-4c28-b819-b24a76c547f3\" (UID: \"29a42f50-8414-4c28-b819-b24a76c547f3\") " Oct 30 19:56:22.036389 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.036335 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca" pod "29a42f50-8414-4c28-b819-b24a76c547f3" (UID: "29a42f50-8414-4c28-b819-b24a76c547f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:22.036514 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.036450 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:22.040069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.040041 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n" pod "29a42f50-8414-4c28-b819-b24a76c547f3" (UID: "29a42f50-8414-4c28-b819-b24a76c547f3"). InnerVolumeSpecName "kube-api-access-rpq8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:22.040194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.040119 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca" pod "29a42f50-8414-4c28-b819-b24a76c547f3" (UID: "29a42f50-8414-4c28-b819-b24a76c547f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:22.138028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.137987 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpq8n\" (UniqueName: \"kubernetes.io/projected/29a42f50-8414-4c28-b819-b24a76c547f3-kube-api-access-rpq8n\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:22.138028 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.138017 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a42f50-8414-4c28-b819-b24a76c547f3-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:22.763395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.763364 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" Oct 30 19:56:22.763573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.763366 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p" event={"ID":"29a42f50-8414-4c28-b819-b24a76c547f3","Type":"ContainerDied","Data":"7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb"} Oct 30 19:56:22.763573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:22.763506 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2d1a4f84eb6fada7fd39b66affd2b88fee1d69f48af66987fbf55a22c07fbb" Oct 30 19:56:37.777780 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:37.777743 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6"] Oct 30 19:56:37.823046 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:37.823009 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a36b7-h8kf6"] Oct 30 19:56:39.670089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:39.670046 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03517dc-81df-4287-a1aa-8e17931037b7" path="/var/lib/kubelet/pods/a03517dc-81df-4287-a1aa-8e17931037b7/volumes" Oct 30 19:56:46.073967 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.073934 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp"] Oct 30 19:56:46.074501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.074487 4789 cpu_manager.go:414] "containerMap: removing container" podUID="29a42f50-8414-4c28-b819-b24a76c547f3" containerName="revoke-keys" Oct 30 19:56:46.074547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.074502 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a42f50-8414-4c28-b819-b24a76c547f3" containerName="revoke-keys" Oct 30 19:56:46.074609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.074600 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a42f50-8414-4c28-b819-b24a76c547f3" containerName="revoke-keys" Oct 30 19:56:46.078048 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.078029 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.133565 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.133534 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp"] Oct 30 19:56:46.175535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.175500 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:56:46.175535 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.175502 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:46.175733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.175551 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:56:46.175733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.175638 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:56:46.194111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.194086 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.194273 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.194136 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdqn\" (UniqueName: \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.194358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.194315 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.295602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.295578 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.295759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.295738 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.295835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.295785 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdqn\" (UniqueName: \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.296313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.296291 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.297815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.297786 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.352833 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.352783 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdqn\" (UniqueName: \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.392298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.392268 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:46.768744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.768716 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp"] Oct 30 19:56:46.771247 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:56:46.771205 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49403bae_5100_47cf_9955_268382ab6550.slice/crio-83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6 WatchSource:0}: Error finding container 83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6: Status 404 returned error can't find the container with id 83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6 Oct 30 19:56:46.948356 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:46.948315 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" event={"ID":"49403bae-5100-47cf-9955-268382ab6550","Type":"ContainerStarted","Data":"83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6"} Oct 30 19:56:47.956707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:47.956678 4789 generic.go:356] "Generic (PLEG): container finished" podID="49403bae-5100-47cf-9955-268382ab6550" containerID="1d2c3a60256e621821db2a2e0a9c5f4349fe6cd591dfad8fbcb00a4f21a79363" exitCode=0 Oct 30 19:56:47.957115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:47.956812 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" event={"ID":"49403bae-5100-47cf-9955-268382ab6550","Type":"ContainerDied","Data":"1d2c3a60256e621821db2a2e0a9c5f4349fe6cd591dfad8fbcb00a4f21a79363"} Oct 30 19:56:48.198694 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:56:48.198651 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:56:48.809233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:48.805977 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp"] Oct 30 19:56:48.848942 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:48.848903 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-660b59-crwfp"] Oct 30 19:56:49.157482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.157460 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:49.333166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.333127 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\") pod \"49403bae-5100-47cf-9955-268382ab6550\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " Oct 30 19:56:49.333379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.333243 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\") pod \"49403bae-5100-47cf-9955-268382ab6550\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " Oct 30 19:56:49.333379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.333306 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\" (UniqueName: \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\") pod \"49403bae-5100-47cf-9955-268382ab6550\" (UID: \"49403bae-5100-47cf-9955-268382ab6550\") " Oct 30 19:56:49.333573 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.333548 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca" pod "49403bae-5100-47cf-9955-268382ab6550" (UID: "49403bae-5100-47cf-9955-268382ab6550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:49.333724 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.333709 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49403bae-5100-47cf-9955-268382ab6550-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:49.335354 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.335327 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca" pod "49403bae-5100-47cf-9955-268382ab6550" (UID: "49403bae-5100-47cf-9955-268382ab6550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:49.335486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.335362 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn" pod "49403bae-5100-47cf-9955-268382ab6550" (UID: "49403bae-5100-47cf-9955-268382ab6550"). InnerVolumeSpecName "kube-api-access-lwdqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:49.435148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.435117 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49403bae-5100-47cf-9955-268382ab6550-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:49.435148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.435142 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdqn\" (UniqueName: \"kubernetes.io/projected/49403bae-5100-47cf-9955-268382ab6550-kube-api-access-lwdqn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:49.668521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.668487 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ed85ae-43a6-43f0-8f8a-d80053838d8a" path="/var/lib/kubelet/pods/20ed85ae-43a6-43f0-8f8a-d80053838d8a/volumes" Oct 30 19:56:49.976911 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.976884 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" Oct 30 19:56:49.977092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.976920 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp" event={"ID":"49403bae-5100-47cf-9955-268382ab6550","Type":"ContainerDied","Data":"83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6"} Oct 30 19:56:49.977092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:49.976970 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83382147ec138924f33cd0ee4ef918a18824040a46c46ed2ba54e0581d4b8ff6" Oct 30 19:56:50.077944 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:50.077916 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg"] Oct 30 19:56:50.080646 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:50.080623 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-7b6aee-fp9rg"] Oct 30 19:56:51.670114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:51.670076 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf853fe2-150f-4ddd-a759-90fd7a6d9b5b" path="/var/lib/kubelet/pods/bf853fe2-150f-4ddd-a759-90fd7a6d9b5b/volumes" Oct 30 19:56:55.854455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:55.854373 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:56:55.854880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:55.854614 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" containerID="cri-o://3cbc45f05e23446b52a837beff6c0d37e5d09805ad9c2cea305dfa33c0727265" gracePeriod=30 Oct 30 19:56:56.722480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.722447 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9"] Oct 30 19:56:56.723091 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.723071 4789 cpu_manager.go:414] "containerMap: removing container" podUID="49403bae-5100-47cf-9955-268382ab6550" containerName="revoke-keys" Oct 30 19:56:56.723234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.723092 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49403bae-5100-47cf-9955-268382ab6550" containerName="revoke-keys" Oct 30 19:56:56.723311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.723240 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="49403bae-5100-47cf-9955-268382ab6550" containerName="revoke-keys" Oct 30 19:56:56.727916 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.727894 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.777417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.777382 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:56:56.777608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.777402 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:56:56.781262 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.781240 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9"] Oct 30 19:56:56.814963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.814910 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x628j\" (UniqueName: \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.815148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.815052 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.815148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.815107 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.823524 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.823501 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:56.823669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.823598 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:56:56.916260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.916204 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x628j\" (UniqueName: \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.916701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.916338 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.916701 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.916381 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.917101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.917063 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.918885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.918863 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:56.977087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:56.977032 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x628j\" (UniqueName: \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\") pod \"maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:57.044498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.044470 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:56:57.439982 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.439951 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9"] Oct 30 19:56:57.441944 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:56:57.441911 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32709e7_0d47_4789_8f17_9e6c732fe370.slice/crio-ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc WatchSource:0}: Error finding container ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc: Status 404 returned error can't find the container with id ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc Oct 30 19:56:57.659442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.659409 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:56:57.659921 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.659886 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" containerID="cri-o://eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51" gracePeriod=30 Oct 30 19:56:57.671504 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.671479 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:56:57.671672 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:57.671650 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" containerID="cri-o://47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096" gracePeriod=30 Oct 30 19:56:58.034555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:58.034447 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" event={"ID":"d32709e7-0d47-4789-8f17-9e6c732fe370","Type":"ContainerStarted","Data":"de986c7ddcbcb8a2d19f17f6167728de18b32ec98c37631aa2a8937d38806cbc"} Oct 30 19:56:58.034555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:58.034516 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" event={"ID":"d32709e7-0d47-4789-8f17-9e6c732fe370","Type":"ContainerStarted","Data":"ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc"} Oct 30 19:56:58.150275 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:58.150202 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" podStartSLOduration=2.150188711 podStartE2EDuration="2.150188711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:58.148491148 +0000 UTC m=+4254.650401646" watchObservedRunningTime="2026-10-30 19:56:58.150188711 +0000 UTC m=+4254.652099197" Oct 30 19:56:58.893229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:58.893182 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:56:59.044041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:59.044001 4789 generic.go:356] "Generic (PLEG): container finished" podID="d32709e7-0d47-4789-8f17-9e6c732fe370" containerID="de986c7ddcbcb8a2d19f17f6167728de18b32ec98c37631aa2a8937d38806cbc" exitCode=0 Oct 30 19:56:59.044473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:59.044053 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" event={"ID":"d32709e7-0d47-4789-8f17-9e6c732fe370","Type":"ContainerDied","Data":"de986c7ddcbcb8a2d19f17f6167728de18b32ec98c37631aa2a8937d38806cbc"} Oct 30 19:56:59.967682 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:59.967650 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc"] Oct 30 19:56:59.971869 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:56:59.971850 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.031042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.030231 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc"] Oct 30 19:57:00.061129 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.061099 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677cv\" (UniqueName: \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.061542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.061332 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.061542 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.061377 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.162204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.162172 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677cv\" (UniqueName: \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.162405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.162305 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.162405 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.162328 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.165400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.164076 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.169620 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.169585 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.221340 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.221271 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-677cv\" (UniqueName: \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.224040 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.224011 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:57:00.262955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.262924 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\") pod \"d32709e7-0d47-4789-8f17-9e6c732fe370\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " Oct 30 19:57:00.263086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.262967 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\" (UniqueName: \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\") pod \"d32709e7-0d47-4789-8f17-9e6c732fe370\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " Oct 30 19:57:00.263086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.263072 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\") pod \"d32709e7-0d47-4789-8f17-9e6c732fe370\" (UID: \"d32709e7-0d47-4789-8f17-9e6c732fe370\") " Oct 30 19:57:00.263311 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.263289 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca" pod "d32709e7-0d47-4789-8f17-9e6c732fe370" (UID: "d32709e7-0d47-4789-8f17-9e6c732fe370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:00.263428 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.263413 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:00.264964 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.264942 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca" pod "d32709e7-0d47-4789-8f17-9e6c732fe370" (UID: "d32709e7-0d47-4789-8f17-9e6c732fe370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:00.265066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.265051 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j" pod "d32709e7-0d47-4789-8f17-9e6c732fe370" (UID: "d32709e7-0d47-4789-8f17-9e6c732fe370"). InnerVolumeSpecName "kube-api-access-x628j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:00.284683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.284661 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:00.364983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.364956 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32709e7-0d47-4789-8f17-9e6c732fe370-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:00.364983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.364983 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x628j\" (UniqueName: \"kubernetes.io/projected/d32709e7-0d47-4789-8f17-9e6c732fe370-kube-api-access-x628j\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:00.681804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:00.681777 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc"] Oct 30 19:57:00.683160 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:00.683134 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8479df94_3dc2_4564_a379_fc6eb31d6a3b.slice/crio-962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4 WatchSource:0}: Error finding container 962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4: Status 404 returned error can't find the container with id 962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4 Oct 30 19:57:01.067229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.063654 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" event={"ID":"d32709e7-0d47-4789-8f17-9e6c732fe370","Type":"ContainerDied","Data":"ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc"} Oct 30 19:57:01.067229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.063714 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed0d2e75c25dc295143c53946bb2f76c733f9da7ef88a3a13a3ad7d004045ffc" Oct 30 19:57:01.067229 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.063783 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9" Oct 30 19:57:01.069595 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.069568 4789 generic.go:356] "Generic (PLEG): container finished" podID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerID="3cbc45f05e23446b52a837beff6c0d37e5d09805ad9c2cea305dfa33c0727265" exitCode=0 Oct 30 19:57:01.069694 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.069644 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerDied","Data":"3cbc45f05e23446b52a837beff6c0d37e5d09805ad9c2cea305dfa33c0727265"} Oct 30 19:57:01.069782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.069706 4789 scope.go:122] "RemoveContainer" containerID="3a62f146deeda15e07fc33102798d3ba639bf393ac751db2bfff71ef3d0cd83a" Oct 30 19:57:01.072189 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.072166 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" event={"ID":"8479df94-3dc2-4564-a379-fc6eb31d6a3b","Type":"ContainerStarted","Data":"5ad5184e992ef65567cc561286b5eba31accb2f13118613c278391a13b849e71"} Oct 30 19:57:01.072315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.072227 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" event={"ID":"8479df94-3dc2-4564-a379-fc6eb31d6a3b","Type":"ContainerStarted","Data":"962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4"} Oct 30 19:57:01.104163 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.104143 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:57:01.174564 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174539 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.174711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174578 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.174711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174629 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.174711 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174676 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.174890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174744 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.174890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174794 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:01.174890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174836 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.175072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174889 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.175072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174919 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.175072 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.174950 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\") pod \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\" (UID: \"9909f7ed-0a52-4248-ac9f-94889dcfd72e\") " Oct 30 19:57:01.175243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175103 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:01.175722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175496 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.175722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175521 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.175722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175559 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:01.175722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175626 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:01.175722 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.175678 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:01.177060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.177040 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:01.177301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.177275 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:01.177403 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.177305 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:01.177604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.177566 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw" pod "9909f7ed-0a52-4248-ac9f-94889dcfd72e" (UID: "9909f7ed-0a52-4248-ac9f-94889dcfd72e"). InnerVolumeSpecName "kube-api-access-l4rhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:01.191022 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.190977 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" podStartSLOduration=2.190961169 podStartE2EDuration="2.190961169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:01.188399411 +0000 UTC m=+4257.690309898" watchObservedRunningTime="2026-10-30 19:57:01.190961169 +0000 UTC m=+4257.692871656" Oct 30 19:57:01.276102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276073 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276097 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276109 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9909f7ed-0a52-4248-ac9f-94889dcfd72e-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276117 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rhw\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-kube-api-access-l4rhw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276126 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276135 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.276323 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:01.276144 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9909f7ed-0a52-4248-ac9f-94889dcfd72e-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.079586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.079556 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" Oct 30 19:57:02.080009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.079586 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l" event={"ID":"9909f7ed-0a52-4248-ac9f-94889dcfd72e","Type":"ContainerDied","Data":"ee802b1107ee4edb94b0896750f516fcd92282e117c044d4cf4143c5384c4349"} Oct 30 19:57:02.080009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.079662 4789 scope.go:122] "RemoveContainer" containerID="3cbc45f05e23446b52a837beff6c0d37e5d09805ad9c2cea305dfa33c0727265" Oct 30 19:57:02.081352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.081325 4789 generic.go:356] "Generic (PLEG): container finished" podID="8479df94-3dc2-4564-a379-fc6eb31d6a3b" containerID="5ad5184e992ef65567cc561286b5eba31accb2f13118613c278391a13b849e71" exitCode=0 Oct 30 19:57:02.081469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.081411 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" event={"ID":"8479df94-3dc2-4564-a379-fc6eb31d6a3b","Type":"ContainerDied","Data":"5ad5184e992ef65567cc561286b5eba31accb2f13118613c278391a13b849e71"} Oct 30 19:57:02.283551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.283519 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:57:02.329141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.329109 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-660b59-openshift-default-58d55dc87c-9wr2l"] Oct 30 19:57:02.959974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.959952 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:57:02.992932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.992911 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:57:02.996156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996129 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996165 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996270 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996199 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996300 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996351 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996331 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996364 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996411 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996446 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996429 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:02.996619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996441 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:02.996619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996459 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996491 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\") pod \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\" (UID: \"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52\") " Oct 30 19:57:02.996782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996644 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:02.996782 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996674 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:02.996929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.996878 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:02.997029 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.997008 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.997108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.997033 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.997108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.997049 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.997108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.997064 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.997108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.997076 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.998457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.998434 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:02.998590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.998572 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:02.998983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.998961 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:02.999049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:02.998975 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7" pod "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" (UID: "1edb2a17-30c6-4ab1-b51a-d687b7dcaa52"). InnerVolumeSpecName "kube-api-access-85lr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:03.090137 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.090109 4789 generic.go:356] "Generic (PLEG): container finished" podID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerID="eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51" exitCode=0 Oct 30 19:57:03.090582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.090157 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" Oct 30 19:57:03.090582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.090196 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" event={"ID":"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52","Type":"ContainerDied","Data":"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51"} Oct 30 19:57:03.090582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.090266 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8" event={"ID":"1edb2a17-30c6-4ab1-b51a-d687b7dcaa52","Type":"ContainerDied","Data":"a92ff4f54841128493c4ec9d71d0d5e7ea437e384094a99a70234988e87df930"} Oct 30 19:57:03.090582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.090297 4789 scope.go:122] "RemoveContainer" containerID="eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51" Oct 30 19:57:03.093681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.093658 4789 generic.go:356] "Generic (PLEG): container finished" podID="efac3a37-19a3-4005-93e1-4496391c982d" containerID="47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096" exitCode=0 Oct 30 19:57:03.093923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.093907 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" Oct 30 19:57:03.094075 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.094035 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerDied","Data":"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096"} Oct 30 19:57:03.094166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.094076 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs" event={"ID":"efac3a37-19a3-4005-93e1-4496391c982d","Type":"ContainerDied","Data":"364462666ebcf3c83fc400026b4a6da576f31640a058eab8ad938f86707a3da1"} Oct 30 19:57:03.097569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097545 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.097684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097609 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.097764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097720 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.097764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097757 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.097888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097820 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.097888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097866 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.098002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097901 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.098002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.097944 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.098123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.098000 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\") pod \"efac3a37-19a3-4005-93e1-4496391c982d\" (UID: \"efac3a37-19a3-4005-93e1-4496391c982d\") " Oct 30 19:57:03.098123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.098027 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:03.098803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.098548 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:03.098803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.098625 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:03.098969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.098891 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:03.099102 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.099067 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:03.099494 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.099477 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efac3a37-19a3-4005-93e1-4496391c982d-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.099669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.099592 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.100114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.100062 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103131 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103110 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103265 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103294 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103310 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85lr7\" (UniqueName: \"kubernetes.io/projected/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-kube-api-access-85lr7\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103326 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103341 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.101683 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:03.103392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.101956 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:03.103791 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103771 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "kube-api-access-xv28l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:03.103877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.103859 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy" pod "efac3a37-19a3-4005-93e1-4496391c982d" (UID: "efac3a37-19a3-4005-93e1-4496391c982d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:03.104626 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.104607 4789 scope.go:122] "RemoveContainer" containerID="eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51" Oct 30 19:57:03.104957 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:03.104930 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51\": container with ID starting with eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51 not found: ID does not exist" containerID="eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51" Oct 30 19:57:03.105060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.104967 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51"} err="failed to get container status \"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51\": rpc error: code = NotFound desc = could not find container \"eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51\": container with ID starting with eda06901cb1b46cbfd2fae6bd730ed6aa291a0a3355365da5d9e039fe88f2a51 not found: ID does not exist" Oct 30 19:57:03.105060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.104989 4789 scope.go:122] "RemoveContainer" containerID="47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096" Oct 30 19:57:03.126391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.126302 4789 scope.go:122] "RemoveContainer" containerID="7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2" Oct 30 19:57:03.136457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.136440 4789 scope.go:122] "RemoveContainer" containerID="47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096" Oct 30 19:57:03.136837 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:03.136817 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096\": container with ID starting with 47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096 not found: ID does not exist" containerID="47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096" Oct 30 19:57:03.136922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.136849 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096"} err="failed to get container status \"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096\": rpc error: code = NotFound desc = could not find container \"47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096\": container with ID starting with 47c8a63a958eeb1fae6c9e604ba3ad19579735f1a5125c7cda19d893c0447096 not found: ID does not exist" Oct 30 19:57:03.136922 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.136874 4789 scope.go:122] "RemoveContainer" containerID="7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2" Oct 30 19:57:03.137124 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:03.137108 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2\": container with ID starting with 7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2 not found: ID does not exist" containerID="7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2" Oct 30 19:57:03.137197 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.137132 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2"} err="failed to get container status \"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2\": rpc error: code = NotFound desc = could not find container \"7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2\": container with ID starting with 7ebf6a825d4b0166454d2d90626884eb00e13216bffdc7637b84b53184eee6c2 not found: ID does not exist" Oct 30 19:57:03.211858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.211824 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efac3a37-19a3-4005-93e1-4496391c982d-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.212080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.212063 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv28l\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-kube-api-access-xv28l\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.212173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.212160 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efac3a37-19a3-4005-93e1-4496391c982d-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.212290 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.212278 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efac3a37-19a3-4005-93e1-4496391c982d-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.238985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.238959 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:03.302379 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.302351 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:57:03.312779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.312750 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\" (UniqueName: \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\") pod \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " Oct 30 19:57:03.312949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.312825 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\") pod \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " Oct 30 19:57:03.312949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.312881 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\") pod \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\" (UID: \"8479df94-3dc2-4564-a379-fc6eb31d6a3b\") " Oct 30 19:57:03.313486 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.313426 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca" pod "8479df94-3dc2-4564-a379-fc6eb31d6a3b" (UID: "8479df94-3dc2-4564-a379-fc6eb31d6a3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:03.315319 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.315296 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca" pod "8479df94-3dc2-4564-a379-fc6eb31d6a3b" (UID: "8479df94-3dc2-4564-a379-fc6eb31d6a3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:03.315435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.315346 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv" pod "8479df94-3dc2-4564-a379-fc6eb31d6a3b" (UID: "8479df94-3dc2-4564-a379-fc6eb31d6a3b"). InnerVolumeSpecName "kube-api-access-677cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:03.316105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.316080 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-h6fw8"] Oct 30 19:57:03.414547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.414498 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-677cv\" (UniqueName: \"kubernetes.io/projected/8479df94-3dc2-4564-a379-fc6eb31d6a3b-kube-api-access-677cv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.414547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.414541 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.414547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.414558 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8479df94-3dc2-4564-a379-fc6eb31d6a3b-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:03.580453 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.580374 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:57:03.588402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.588376 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-7b6aee-openshift-default-6fddfcd74f-rljcs"] Oct 30 19:57:03.671430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.671397 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" path="/var/lib/kubelet/pods/1edb2a17-30c6-4ab1-b51a-d687b7dcaa52/volumes" Oct 30 19:57:03.671958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.671936 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" path="/var/lib/kubelet/pods/9909f7ed-0a52-4248-ac9f-94889dcfd72e/volumes" Oct 30 19:57:03.672455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.672436 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efac3a37-19a3-4005-93e1-4496391c982d" path="/var/lib/kubelet/pods/efac3a37-19a3-4005-93e1-4496391c982d/volumes" Oct 30 19:57:03.913288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:03.912279 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" containerID="cri-o://fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf" gracePeriod=25 Oct 30 19:57:04.125391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:04.125361 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" Oct 30 19:57:04.125875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:04.125393 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc" event={"ID":"8479df94-3dc2-4564-a379-fc6eb31d6a3b","Type":"ContainerDied","Data":"962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4"} Oct 30 19:57:04.125875 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:04.125439 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962dcfade490ade520a84f91077b62c1ecfb5910fed43aad31a8c2a15718e1c4" Oct 30 19:57:04.216883 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:04.216845 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:07.534543 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:07.534504 4789 scope.go:122] "RemoveContainer" containerID="a837d1a8a6008cb664bca95eb904f95047f9b218d365c6151a37438b042c97da" Oct 30 19:57:08.122233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:08.121503 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:57:08.227251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:08.227184 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:57:10.521461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:10.521423 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:57:12.360847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:12.359006 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": read tcp 10.132.0.2:36618->10.132.1.44:15021: read: connection reset by peer" start-of-body= Oct 30 19:57:12.360847 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:12.359079 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": read tcp 10.132.0.2:36618->10.132.1.44:15021: read: connection reset by peer" Oct 30 19:57:13.122498 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.122446 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" podUID="edc329f6-8646-4186-a57d-aa3b58103119" containerName="payload-processing" containerID="cri-o://552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63" gracePeriod=25 Oct 30 19:57:13.190648 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.190620 4789 generic.go:356] "Generic (PLEG): container finished" podID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerID="3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7" exitCode=0 Oct 30 19:57:13.190781 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.190701 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerDied","Data":"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7"} Oct 30 19:57:13.191121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.191105 4789 scope.go:122] "RemoveContainer" containerID="3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7" Oct 30 19:57:13.192969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.192936 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:13.193062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.193025 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:13.193106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.193097 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:13.227636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.227606 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" podUID="12b084f6-bd1c-4274-bac7-be10962e4c7c" containerName="payload-pre-processing" containerID="cri-o://d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12" gracePeriod=25 Oct 30 19:57:13.453227 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.451435 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:57:13.534599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.532892 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\" (UniqueName: \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\") pod \"edc329f6-8646-4186-a57d-aa3b58103119\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " Oct 30 19:57:13.534599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.532937 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\") pod \"edc329f6-8646-4186-a57d-aa3b58103119\" (UID: \"edc329f6-8646-4186-a57d-aa3b58103119\") " Oct 30 19:57:13.534599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.533567 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume" pod "edc329f6-8646-4186-a57d-aa3b58103119" (UID: "edc329f6-8646-4186-a57d-aa3b58103119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:13.535684 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.535656 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv" pod "edc329f6-8646-4186-a57d-aa3b58103119" (UID: "edc329f6-8646-4186-a57d-aa3b58103119"). InnerVolumeSpecName "kube-api-access-m6bcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:13.566990 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.566966 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:57:13.634594 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.634558 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\" (UniqueName: \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\") pod \"12b084f6-bd1c-4274-bac7-be10962e4c7c\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " Oct 30 19:57:13.634784 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.634677 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\") pod \"12b084f6-bd1c-4274-bac7-be10962e4c7c\" (UID: \"12b084f6-bd1c-4274-bac7-be10962e4c7c\") " Oct 30 19:57:13.635147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.635120 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6bcv\" (UniqueName: \"kubernetes.io/projected/edc329f6-8646-4186-a57d-aa3b58103119-kube-api-access-m6bcv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:13.635314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.635150 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edc329f6-8646-4186-a57d-aa3b58103119-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:13.635314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.635298 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume" pod "12b084f6-bd1c-4274-bac7-be10962e4c7c" (UID: "12b084f6-bd1c-4274-bac7-be10962e4c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:13.637260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.637229 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs" pod "12b084f6-bd1c-4274-bac7-be10962e4c7c" (UID: "12b084f6-bd1c-4274-bac7-be10962e4c7c"). InnerVolumeSpecName "kube-api-access-bjbbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:13.735961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.735924 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b084f6-bd1c-4274-bac7-be10962e4c7c-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:13.735961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:13.735962 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbbs\" (UniqueName: \"kubernetes.io/projected/12b084f6-bd1c-4274-bac7-be10962e4c7c-kube-api-access-bjbbs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:14.217231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.216331 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:57:14.228233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.223503 4789 generic.go:356] "Generic (PLEG): container finished" podID="12b084f6-bd1c-4274-bac7-be10962e4c7c" containerID="d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12" exitCode=0 Oct 30 19:57:14.228233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.223800 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" event={"ID":"12b084f6-bd1c-4274-bac7-be10962e4c7c","Type":"ContainerDied","Data":"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12"} Oct 30 19:57:14.228233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.223871 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" event={"ID":"12b084f6-bd1c-4274-bac7-be10962e4c7c","Type":"ContainerDied","Data":"5239cffa21d079e4fe82fa64c8fa98ef251fa29bc1e059eca3aacb116b2c8d67"} Oct 30 19:57:14.228233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.223920 4789 scope.go:122] "RemoveContainer" containerID="d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12" Oct 30 19:57:14.228233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.224102 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d" Oct 30 19:57:14.231903 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.231869 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerStarted","Data":"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45"} Oct 30 19:57:14.233546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.233516 4789 generic.go:356] "Generic (PLEG): container finished" podID="edc329f6-8646-4186-a57d-aa3b58103119" containerID="552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63" exitCode=0 Oct 30 19:57:14.233658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.233555 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" Oct 30 19:57:14.233658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.233583 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" event={"ID":"edc329f6-8646-4186-a57d-aa3b58103119","Type":"ContainerDied","Data":"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63"} Oct 30 19:57:14.233658 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.233624 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph" event={"ID":"edc329f6-8646-4186-a57d-aa3b58103119","Type":"ContainerDied","Data":"40671b5a72ef64708e586ecdff21a36a16a09b31653eeffc6952fb14c48556ed"} Oct 30 19:57:14.241147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.241122 4789 scope.go:122] "RemoveContainer" containerID="d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12" Oct 30 19:57:14.241423 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:14.241398 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12\": container with ID starting with d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12 not found: ID does not exist" containerID="d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12" Oct 30 19:57:14.241495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.241434 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12"} err="failed to get container status \"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12\": rpc error: code = NotFound desc = could not find container \"d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12\": container with ID starting with d485a5ae1aee54cce017f595b5394258a0185f2ccf3ffe76679e7b75277b6d12 not found: ID does not exist" Oct 30 19:57:14.241495 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.241457 4789 scope.go:122] "RemoveContainer" containerID="552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63" Oct 30 19:57:14.250436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.250420 4789 scope.go:122] "RemoveContainer" containerID="552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63" Oct 30 19:57:14.250705 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:14.250686 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63\": container with ID starting with 552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63 not found: ID does not exist" containerID="552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63" Oct 30 19:57:14.250759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.250711 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63"} err="failed to get container status \"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63\": rpc error: code = NotFound desc = could not find container \"552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63\": container with ID starting with 552836c2749b35b9885d4000d8a1cea6e75c8159bf832da47127291dade01f63 not found: ID does not exist" Oct 30 19:57:14.385679 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.385646 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:57:14.390016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.389989 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-312161-856d75cdd7-dqr8d"] Oct 30 19:57:14.495289 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.492158 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:57:14.520414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.512784 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-312161-6f68889998-rghph"] Oct 30 19:57:14.772816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.772736 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:57:14.773066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.773040 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 19:57:14.773172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:14.773098 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 19:57:15.522725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.522687 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" podUID="a59688af-426d-4856-b0a2-8fd334be746a" containerName="payload-pre-processing" containerID="cri-o://d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c" gracePeriod=25 Oct 30 19:57:15.669199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.669159 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b084f6-bd1c-4274-bac7-be10962e4c7c" path="/var/lib/kubelet/pods/12b084f6-bd1c-4274-bac7-be10962e4c7c/volumes" Oct 30 19:57:15.669696 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.669653 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc329f6-8646-4186-a57d-aa3b58103119" path="/var/lib/kubelet/pods/edc329f6-8646-4186-a57d-aa3b58103119/volumes" Oct 30 19:57:15.773242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.773140 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 19:57:15.773242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.773203 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 19:57:15.837947 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.837921 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:57:15.959532 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.959497 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\") pod \"a59688af-426d-4856-b0a2-8fd334be746a\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " Oct 30 19:57:15.959835 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.959795 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\") pod \"a59688af-426d-4856-b0a2-8fd334be746a\" (UID: \"a59688af-426d-4856-b0a2-8fd334be746a\") " Oct 30 19:57:15.960493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.960464 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume" pod "a59688af-426d-4856-b0a2-8fd334be746a" (UID: "a59688af-426d-4856-b0a2-8fd334be746a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:15.962757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:15.962732 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw" pod "a59688af-426d-4856-b0a2-8fd334be746a" (UID: "a59688af-426d-4856-b0a2-8fd334be746a"). InnerVolumeSpecName "kube-api-access-js2bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:16.061002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.060923 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2bw\" (UniqueName: \"kubernetes.io/projected/a59688af-426d-4856-b0a2-8fd334be746a-kube-api-access-js2bw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:16.061002 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.060948 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a59688af-426d-4856-b0a2-8fd334be746a-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:16.249265 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.249198 4789 generic.go:356] "Generic (PLEG): container finished" podID="a59688af-426d-4856-b0a2-8fd334be746a" containerID="d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c" exitCode=0 Oct 30 19:57:16.249437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.249273 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" Oct 30 19:57:16.249437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.249237 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" event={"ID":"a59688af-426d-4856-b0a2-8fd334be746a","Type":"ContainerDied","Data":"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c"} Oct 30 19:57:16.249437 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.249407 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6" event={"ID":"a59688af-426d-4856-b0a2-8fd334be746a","Type":"ContainerDied","Data":"39045af7242a77ad6f6be6d451972f698a51fee51a125cd9afa97348ea044c6c"} Oct 30 19:57:16.249557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.249435 4789 scope.go:122] "RemoveContainer" containerID="d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c" Oct 30 19:57:16.259760 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.259741 4789 scope.go:122] "RemoveContainer" containerID="d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c" Oct 30 19:57:16.260017 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:16.259994 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c\": container with ID starting with d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c not found: ID does not exist" containerID="d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c" Oct 30 19:57:16.260115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.260023 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c"} err="failed to get container status \"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c\": rpc error: code = NotFound desc = could not find container \"d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c\": container with ID starting with d660c6d50c30064ef81f37d55e3641723c06855370580cc42d6db8e55554077c not found: ID does not exist" Oct 30 19:57:16.418478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.418373 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:57:16.462675 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.462636 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-2186d8-79f8986bc-9qpr6"] Oct 30 19:57:16.773700 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.773667 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 19:57:16.774086 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:16.773733 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 19:57:17.668955 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:17.668918 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59688af-426d-4856-b0a2-8fd334be746a" path="/var/lib/kubelet/pods/a59688af-426d-4856-b0a2-8fd334be746a/volumes" Oct 30 19:57:17.773467 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:17.773432 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 19:57:17.773662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:17.773503 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 19:57:18.231464 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:18.231384 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:57:18.773591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:18.773551 4789 patch_prober.go:29] interesting pod/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Oct 30 19:57:18.773771 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:18.773618 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Oct 30 19:57:19.777186 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:19.777156 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:57:19.777755 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:19.777731 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:57:19.778409 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:19.778389 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 19:57:19.921562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:19.921525 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc"] Oct 30 19:57:19.924370 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:19.924351 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-2186d8-wzsmc"] Oct 30 19:57:21.670864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:21.670828 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8479df94-3dc2-4564-a379-fc6eb31d6a3b" path="/var/lib/kubelet/pods/8479df94-3dc2-4564-a379-fc6eb31d6a3b/volumes" Oct 30 19:57:24.205430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.205405 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:57:24.310733 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.310702 4789 generic.go:356] "Generic (PLEG): container finished" podID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerID="fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf" exitCode=0 Oct 30 19:57:24.310879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.310742 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" Oct 30 19:57:24.310879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.310749 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" event={"ID":"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62","Type":"ContainerDied","Data":"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf"} Oct 30 19:57:24.310879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.310795 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" event={"ID":"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62","Type":"ContainerDied","Data":"7d9f9cc9e6350cac9781c47a756874a318534ccbd25bb19291d89235491ab173"} Oct 30 19:57:24.310879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.310817 4789 scope.go:122] "RemoveContainer" containerID="fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf" Oct 30 19:57:24.324383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.324355 4789 scope.go:122] "RemoveContainer" containerID="fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf" Oct 30 19:57:24.324773 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:24.324739 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf\": container with ID starting with fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf not found: ID does not exist" containerID="fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf" Oct 30 19:57:24.324870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.324784 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf"} err="failed to get container status \"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf\": rpc error: code = NotFound desc = could not find container \"fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf\": container with ID starting with fbeba1fd2f9f26cc762549100a7700b3ae0bbee7c6d1de06cf4cb43b732e8bbf not found: ID does not exist" Oct 30 19:57:24.371317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.371283 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\") pod \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " Oct 30 19:57:24.371480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.371448 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\" (UniqueName: \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\") pod \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\" (UID: \"15d1d59f-1e56-41c0-ade3-10fa0ffe5e62\") " Oct 30 19:57:24.371741 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.371719 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume" pod "15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" (UID: "15d1d59f-1e56-41c0-ade3-10fa0ffe5e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:24.371891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.371873 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:24.373571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.373554 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz" pod "15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" (UID: "15d1d59f-1e56-41c0-ade3-10fa0ffe5e62"). InnerVolumeSpecName "kube-api-access-8p8gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:24.472794 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.472764 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p8gz\" (UniqueName: \"kubernetes.io/projected/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62-kube-api-access-8p8gz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:24.730560 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.730476 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:57:24.734823 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:24.734795 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z"] Oct 30 19:57:25.198673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:25.198630 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-db06cf-756777f787-x6s8z" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.225:50052\" within 1s: context deadline exceeded" Oct 30 19:57:25.198843 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:25.198742 4789 logging.go:55] [core] [Channel #6590 SubChannel #6591]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.225:50052", ServerName: "10.132.0.225:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.225:50052: operation was canceled" Oct 30 19:57:25.668400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:25.668307 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" path="/var/lib/kubelet/pods/15d1d59f-1e56-41c0-ade3-10fa0ffe5e62/volumes" Oct 30 19:57:28.702148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702110 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx"] Oct 30 19:57:28.702820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702799 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8479df94-3dc2-4564-a379-fc6eb31d6a3b" containerName="revoke-keys" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702822 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8479df94-3dc2-4564-a379-fc6eb31d6a3b" containerName="revoke-keys" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702862 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a59688af-426d-4856-b0a2-8fd334be746a" containerName="payload-pre-processing" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702872 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59688af-426d-4856-b0a2-8fd334be746a" containerName="payload-pre-processing" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702888 4789 cpu_manager.go:414] "containerMap: removing container" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702896 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702905 4789 cpu_manager.go:414] "containerMap: removing container" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.702919 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702912 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702928 4789 cpu_manager.go:414] "containerMap: removing container" podUID="edc329f6-8646-4186-a57d-aa3b58103119" containerName="payload-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702935 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc329f6-8646-4186-a57d-aa3b58103119" containerName="payload-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702954 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d32709e7-0d47-4789-8f17-9e6c732fe370" containerName="revoke-keys" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702962 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32709e7-0d47-4789-8f17-9e6c732fe370" containerName="revoke-keys" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702972 4789 cpu_manager.go:414] "containerMap: removing container" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702979 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.702999 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703007 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703023 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703031 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703042 4789 cpu_manager.go:414] "containerMap: removing container" podUID="12b084f6-bd1c-4274-bac7-be10962e4c7c" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703049 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b084f6-bd1c-4274-bac7-be10962e4c7c" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703069 4789 cpu_manager.go:414] "containerMap: removing container" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703077 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703221 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8479df94-3dc2-4564-a379-fc6eb31d6a3b" containerName="revoke-keys" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703242 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59688af-426d-4856-b0a2-8fd334be746a" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703253 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb2a17-30c6-4ab1-b51a-d687b7dcaa52" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703262 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703273 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc329f6-8646-4186-a57d-aa3b58103119" containerName="payload-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703286 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32709e7-0d47-4789-8f17-9e6c732fe370" containerName="revoke-keys" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703297 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="efac3a37-19a3-4005-93e1-4496391c982d" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703308 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703321 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9909f7ed-0a52-4248-ac9f-94889dcfd72e" containerName="istio-proxy" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703330 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b084f6-bd1c-4274-bac7-be10962e4c7c" containerName="payload-pre-processing" Oct 30 19:57:28.703329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.703340 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d1d59f-1e56-41c0-ade3-10fa0ffe5e62" containerName="payload-pre-processing" Oct 30 19:57:28.708480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.708459 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.764442 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.764416 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx"] Oct 30 19:57:28.807062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.807035 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:28.807179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.807066 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:57:28.807179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.807079 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:57:28.807179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.807153 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:57:28.820975 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.820952 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.821081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.820988 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pz7v\" (UniqueName: \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.821081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.821043 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.922093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.922061 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.922245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.922100 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pz7v\" (UniqueName: \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.922245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.922174 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.922984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.922960 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.924562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.924541 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:28.989411 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:28.989358 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pz7v\" (UniqueName: \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:29.022572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:29.022551 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:29.405503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:29.405478 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx"] Oct 30 19:57:29.408660 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:29.408629 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3b2342_3a74_42dc_b63f_50dd13777d73.slice/crio-e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e WatchSource:0}: Error finding container e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e: Status 404 returned error can't find the container with id e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e Oct 30 19:57:30.355584 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:30.355554 4789 generic.go:356] "Generic (PLEG): container finished" podID="df3b2342-3a74-42dc-b63f-50dd13777d73" containerID="51d37c786d4e453f8c5802b7ec6bbdfd268046240a33cdb932c4b5c2fc6243f0" exitCode=0 Oct 30 19:57:30.355943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:30.355625 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" event={"ID":"df3b2342-3a74-42dc-b63f-50dd13777d73","Type":"ContainerDied","Data":"51d37c786d4e453f8c5802b7ec6bbdfd268046240a33cdb932c4b5c2fc6243f0"} Oct 30 19:57:30.355943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:30.355676 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" event={"ID":"df3b2342-3a74-42dc-b63f-50dd13777d73","Type":"ContainerStarted","Data":"e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e"} Oct 30 19:57:31.517376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.517348 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:31.651274 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.651174 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\" (UniqueName: \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\") pod \"df3b2342-3a74-42dc-b63f-50dd13777d73\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " Oct 30 19:57:31.651459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.651270 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\") pod \"df3b2342-3a74-42dc-b63f-50dd13777d73\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " Oct 30 19:57:31.651459 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.651383 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\") pod \"df3b2342-3a74-42dc-b63f-50dd13777d73\" (UID: \"df3b2342-3a74-42dc-b63f-50dd13777d73\") " Oct 30 19:57:31.651728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.651699 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca" pod "df3b2342-3a74-42dc-b63f-50dd13777d73" (UID: "df3b2342-3a74-42dc-b63f-50dd13777d73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:31.651858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.651818 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:31.653300 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.653274 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca" pod "df3b2342-3a74-42dc-b63f-50dd13777d73" (UID: "df3b2342-3a74-42dc-b63f-50dd13777d73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:31.653414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.653398 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v" pod "df3b2342-3a74-42dc-b63f-50dd13777d73" (UID: "df3b2342-3a74-42dc-b63f-50dd13777d73"). InnerVolumeSpecName "kube-api-access-9pz7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:31.753192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.753166 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df3b2342-3a74-42dc-b63f-50dd13777d73-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:31.753192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:31.753198 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pz7v\" (UniqueName: \"kubernetes.io/projected/df3b2342-3a74-42dc-b63f-50dd13777d73-kube-api-access-9pz7v\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:32.369375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:32.369347 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" Oct 30 19:57:32.369375 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:32.369361 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx" event={"ID":"df3b2342-3a74-42dc-b63f-50dd13777d73","Type":"ContainerDied","Data":"e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e"} Oct 30 19:57:32.369599 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:32.369405 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e734d264e1454b9d2c215d7a85ed084211327a1a37e4fbade3a8d593ffb90c9e" Oct 30 19:57:36.050974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:36.050935 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp"] Oct 30 19:57:36.097138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:36.097093 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-db06cf-xppnp"] Oct 30 19:57:37.668687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:37.668649 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49403bae-5100-47cf-9955-268382ab6550" path="/var/lib/kubelet/pods/49403bae-5100-47cf-9955-268382ab6550/volumes" Oct 30 19:57:40.340800 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:40.340760 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:57:40.438558 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:40.438524 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:57:42.306905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:42.306868 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p"] Oct 30 19:57:42.353285 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:42.353257 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fe7634-htl2p"] Oct 30 19:57:43.668123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:43.668089 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a42f50-8414-4c28-b819-b24a76c547f3" path="/var/lib/kubelet/pods/29a42f50-8414-4c28-b819-b24a76c547f3/volumes" Oct 30 19:57:44.405972 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.405937 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w"] Oct 30 19:57:44.406484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.406468 4789 cpu_manager.go:414] "containerMap: removing container" podUID="df3b2342-3a74-42dc-b63f-50dd13777d73" containerName="revoke-keys" Oct 30 19:57:44.406544 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.406486 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3b2342-3a74-42dc-b63f-50dd13777d73" containerName="revoke-keys" Oct 30 19:57:44.406591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.406581 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3b2342-3a74-42dc-b63f-50dd13777d73" containerName="revoke-keys" Oct 30 19:57:44.409912 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.409895 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.461230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.461179 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:57:44.477202 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.477177 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:57:44.478686 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.478657 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.478820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.478709 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmpwd\" (UniqueName: \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.478820 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.478785 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.510149 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.510126 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w"] Oct 30 19:57:44.518268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.518251 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:44.518341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.518306 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:57:44.579422 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.579390 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.579536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.579444 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmpwd\" (UniqueName: \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.579536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.579468 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.580066 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.580038 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.581795 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.581776 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.637060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.637032 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmpwd\" (UniqueName: \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:44.723328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:44.723299 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:45.095977 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.095954 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w"] Oct 30 19:57:45.098109 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:45.098078 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc721f35_c500_4d52_aa3d_49ebca41a9fb.slice/crio-53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621 WatchSource:0}: Error finding container 53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621: Status 404 returned error can't find the container with id 53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621 Oct 30 19:57:45.341952 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.341916 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" podUID="898fd678-caa5-4550-8086-7ae7b848e63c" containerName="payload-processing" containerID="cri-o://04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189" gracePeriod=25 Oct 30 19:57:45.439444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.439390 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" podUID="9ed8748b-7c88-4222-8225-51a52ab4c593" containerName="payload-pre-processing" containerID="cri-o://07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6" gracePeriod=25 Oct 30 19:57:45.455499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.455465 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" event={"ID":"bc721f35-c500-4d52-aa3d-49ebca41a9fb","Type":"ContainerStarted","Data":"53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621"} Oct 30 19:57:45.618081 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.618044 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:57:45.689536 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.689508 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\" (UniqueName: \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\") pod \"898fd678-caa5-4550-8086-7ae7b848e63c\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " Oct 30 19:57:45.689727 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.689634 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\") pod \"898fd678-caa5-4550-8086-7ae7b848e63c\" (UID: \"898fd678-caa5-4550-8086-7ae7b848e63c\") " Oct 30 19:57:45.690016 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.689994 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume" pod "898fd678-caa5-4550-8086-7ae7b848e63c" (UID: "898fd678-caa5-4550-8086-7ae7b848e63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:45.692009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.691969 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh" pod "898fd678-caa5-4550-8086-7ae7b848e63c" (UID: "898fd678-caa5-4550-8086-7ae7b848e63c"). InnerVolumeSpecName "kube-api-access-svpbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:45.753170 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.753136 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:57:45.791603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.791570 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpbh\" (UniqueName: \"kubernetes.io/projected/898fd678-caa5-4550-8086-7ae7b848e63c-kube-api-access-svpbh\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:45.791603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.791605 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898fd678-caa5-4550-8086-7ae7b848e63c-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:45.893106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.893011 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\") pod \"9ed8748b-7c88-4222-8225-51a52ab4c593\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " Oct 30 19:57:45.893292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.893161 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\" (UniqueName: \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\") pod \"9ed8748b-7c88-4222-8225-51a52ab4c593\" (UID: \"9ed8748b-7c88-4222-8225-51a52ab4c593\") " Oct 30 19:57:45.895824 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.895797 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume" pod "9ed8748b-7c88-4222-8225-51a52ab4c593" (UID: "9ed8748b-7c88-4222-8225-51a52ab4c593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:45.896064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.896038 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5" pod "9ed8748b-7c88-4222-8225-51a52ab4c593" (UID: "9ed8748b-7c88-4222-8225-51a52ab4c593"). InnerVolumeSpecName "kube-api-access-fxbg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:45.994121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.994091 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed8748b-7c88-4222-8225-51a52ab4c593-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:45.994121 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:45.994117 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbg5\" (UniqueName: \"kubernetes.io/projected/9ed8748b-7c88-4222-8225-51a52ab4c593-kube-api-access-fxbg5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:46.463183 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.463148 4789 generic.go:356] "Generic (PLEG): container finished" podID="bc721f35-c500-4d52-aa3d-49ebca41a9fb" containerID="b191ff4c5a88963d25307be39b4077bd92caca13ae61fede2ced07ddd7333cc7" exitCode=0 Oct 30 19:57:46.463376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.463224 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" event={"ID":"bc721f35-c500-4d52-aa3d-49ebca41a9fb","Type":"ContainerDied","Data":"b191ff4c5a88963d25307be39b4077bd92caca13ae61fede2ced07ddd7333cc7"} Oct 30 19:57:46.464506 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.464482 4789 generic.go:356] "Generic (PLEG): container finished" podID="9ed8748b-7c88-4222-8225-51a52ab4c593" containerID="07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6" exitCode=0 Oct 30 19:57:46.464636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.464532 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" Oct 30 19:57:46.464636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.464557 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" event={"ID":"9ed8748b-7c88-4222-8225-51a52ab4c593","Type":"ContainerDied","Data":"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6"} Oct 30 19:57:46.464636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.464598 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s" event={"ID":"9ed8748b-7c88-4222-8225-51a52ab4c593","Type":"ContainerDied","Data":"97cd84d7c520f2d16d671a3e7b1a91f9283cdddf3bc3599e00b43e378995996a"} Oct 30 19:57:46.464636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.464626 4789 scope.go:122] "RemoveContainer" containerID="07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6" Oct 30 19:57:46.466052 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.466034 4789 generic.go:356] "Generic (PLEG): container finished" podID="898fd678-caa5-4550-8086-7ae7b848e63c" containerID="04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189" exitCode=0 Oct 30 19:57:46.466169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.466067 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" event={"ID":"898fd678-caa5-4550-8086-7ae7b848e63c","Type":"ContainerDied","Data":"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189"} Oct 30 19:57:46.466169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.466112 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" Oct 30 19:57:46.466169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.466121 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff" event={"ID":"898fd678-caa5-4550-8086-7ae7b848e63c","Type":"ContainerDied","Data":"eaa5ca79ba52ff78c0f6e3dbd71987b533428df7eedcea21e657a600c4c2e0b6"} Oct 30 19:57:46.475394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.475327 4789 scope.go:122] "RemoveContainer" containerID="07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6" Oct 30 19:57:46.475596 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:46.475573 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6\": container with ID starting with 07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6 not found: ID does not exist" containerID="07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6" Oct 30 19:57:46.475639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.475603 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6"} err="failed to get container status \"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6\": rpc error: code = NotFound desc = could not find container \"07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6\": container with ID starting with 07dbe006fb10cb393fab79b4f45e201cb93130f8868cbbde5677e72e0b639db6 not found: ID does not exist" Oct 30 19:57:46.475639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.475620 4789 scope.go:122] "RemoveContainer" containerID="04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189" Oct 30 19:57:46.484238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.484200 4789 scope.go:122] "RemoveContainer" containerID="04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189" Oct 30 19:57:46.484510 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:46.484490 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189\": container with ID starting with 04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189 not found: ID does not exist" containerID="04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189" Oct 30 19:57:46.484579 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.484519 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189"} err="failed to get container status \"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189\": rpc error: code = NotFound desc = could not find container \"04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189\": container with ID starting with 04c97a8629de82300ee6839d382005ad41257ab80985642b2e7c5c504c251189 not found: ID does not exist" Oct 30 19:57:46.681125 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.681096 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:57:46.687706 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.687685 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d6f58f-76bddc88-kf4ff"] Oct 30 19:57:46.802490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.802417 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:57:46.809702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:46.809673 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d6f58f-5b9688f479-8lz6s"] Oct 30 19:57:47.618266 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.618239 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:47.668060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.668031 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898fd678-caa5-4550-8086-7ae7b848e63c" path="/var/lib/kubelet/pods/898fd678-caa5-4550-8086-7ae7b848e63c/volumes" Oct 30 19:57:47.668539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.668507 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed8748b-7c88-4222-8225-51a52ab4c593" path="/var/lib/kubelet/pods/9ed8748b-7c88-4222-8225-51a52ab4c593/volumes" Oct 30 19:57:47.708574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.708546 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\") pod \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " Oct 30 19:57:47.708708 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.708610 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\") pod \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " Oct 30 19:57:47.708826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.708800 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\" (UniqueName: \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\") pod \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\" (UID: \"bc721f35-c500-4d52-aa3d-49ebca41a9fb\") " Oct 30 19:57:47.708973 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.708947 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca" pod "bc721f35-c500-4d52-aa3d-49ebca41a9fb" (UID: "bc721f35-c500-4d52-aa3d-49ebca41a9fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:47.709466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.709308 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:47.710876 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.710853 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd" pod "bc721f35-c500-4d52-aa3d-49ebca41a9fb" (UID: "bc721f35-c500-4d52-aa3d-49ebca41a9fb"). InnerVolumeSpecName "kube-api-access-bmpwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:47.711140 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.711122 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca" pod "bc721f35-c500-4d52-aa3d-49ebca41a9fb" (UID: "bc721f35-c500-4d52-aa3d-49ebca41a9fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:47.810400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.810333 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc721f35-c500-4d52-aa3d-49ebca41a9fb-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:47.810400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:47.810359 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmpwd\" (UniqueName: \"kubernetes.io/projected/bc721f35-c500-4d52-aa3d-49ebca41a9fb-kube-api-access-bmpwd\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:48.196835 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:48.196802 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:57:48.482190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:48.482115 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" Oct 30 19:57:48.482190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:48.482158 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w" event={"ID":"bc721f35-c500-4d52-aa3d-49ebca41a9fb","Type":"ContainerDied","Data":"53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621"} Oct 30 19:57:48.482392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:48.482204 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bd0be79e18bf3147bb6c2899021057c0dac75e3fc41a2b0830e24825311621" Oct 30 19:57:50.807487 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:50.807452 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:57:50.807929 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:50.807705 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" containerID="cri-o://748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f" gracePeriod=30 Oct 30 19:57:51.657969 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.657935 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp"] Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659182 4789 cpu_manager.go:414] "containerMap: removing container" podUID="898fd678-caa5-4550-8086-7ae7b848e63c" containerName="payload-processing" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659249 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898fd678-caa5-4550-8086-7ae7b848e63c" containerName="payload-processing" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659332 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9ed8748b-7c88-4222-8225-51a52ab4c593" containerName="payload-pre-processing" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659342 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed8748b-7c88-4222-8225-51a52ab4c593" containerName="payload-pre-processing" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659362 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bc721f35-c500-4d52-aa3d-49ebca41a9fb" containerName="revoke-keys" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659369 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc721f35-c500-4d52-aa3d-49ebca41a9fb" containerName="revoke-keys" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659712 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed8748b-7c88-4222-8225-51a52ab4c593" containerName="payload-pre-processing" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659737 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc721f35-c500-4d52-aa3d-49ebca41a9fb" containerName="revoke-keys" Oct 30 19:57:51.660943 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.659762 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="898fd678-caa5-4550-8086-7ae7b848e63c" containerName="payload-processing" Oct 30 19:57:51.670158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.670131 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.718231 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.717312 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp"] Oct 30 19:57:51.722268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.721082 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:57:51.722268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.721251 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:57:51.749886 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.749853 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c6c\" (UniqueName: \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.750107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.750081 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.750251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.750229 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.765037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.765013 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:51.765037 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.765032 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:57:51.851918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.851868 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.852439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.851971 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c6c\" (UniqueName: \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.852439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.852132 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.852803 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.852784 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.854932 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.854902 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.912377 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.912305 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c6c\" (UniqueName: \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:51.986268 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:51.986234 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:52.373357 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:52.373319 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp"] Oct 30 19:57:52.377033 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:52.377004 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ada151_486a_425b_89d5_f21ad262e7f4.slice/crio-dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666 WatchSource:0}: Error finding container dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666: Status 404 returned error can't find the container with id dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666 Oct 30 19:57:52.512180 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:52.512147 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" event={"ID":"c5ada151-486a-425b-89d5-f21ad262e7f4","Type":"ContainerStarted","Data":"dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666"} Oct 30 19:57:52.633301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:52.633230 4789 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:57:52.633301 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:52.633283 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:57:53.521546 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:53.521511 4789 generic.go:356] "Generic (PLEG): container finished" podID="c5ada151-486a-425b-89d5-f21ad262e7f4" containerID="b281eba6ce91dbe4e5d29900dddd7223852b29a3168166973ad24f246ebdb5e2" exitCode=0 Oct 30 19:57:53.522042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:53.521597 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" event={"ID":"c5ada151-486a-425b-89d5-f21ad262e7f4","Type":"ContainerDied","Data":"b281eba6ce91dbe4e5d29900dddd7223852b29a3168166973ad24f246ebdb5e2"} Oct 30 19:57:54.672173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.672150 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:54.782089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.782030 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\") pod \"c5ada151-486a-425b-89d5-f21ad262e7f4\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " Oct 30 19:57:54.782089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.782068 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\") pod \"c5ada151-486a-425b-89d5-f21ad262e7f4\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " Oct 30 19:57:54.782320 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.782192 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\" (UniqueName: \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\") pod \"c5ada151-486a-425b-89d5-f21ad262e7f4\" (UID: \"c5ada151-486a-425b-89d5-f21ad262e7f4\") " Oct 30 19:57:54.782397 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.782372 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca" pod "c5ada151-486a-425b-89d5-f21ad262e7f4" (UID: "c5ada151-486a-425b-89d5-f21ad262e7f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:54.782540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.782521 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:54.784248 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.784225 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca" pod "c5ada151-486a-425b-89d5-f21ad262e7f4" (UID: "c5ada151-486a-425b-89d5-f21ad262e7f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:54.784344 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.784226 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c" pod "c5ada151-486a-425b-89d5-f21ad262e7f4" (UID: "c5ada151-486a-425b-89d5-f21ad262e7f4"). InnerVolumeSpecName "kube-api-access-h4c6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:54.883087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.883053 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4c6c\" (UniqueName: \"kubernetes.io/projected/c5ada151-486a-425b-89d5-f21ad262e7f4-kube-api-access-h4c6c\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:54.883087 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:54.883083 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ada151-486a-425b-89d5-f21ad262e7f4-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:55.059908 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.059827 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:57:55.320526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.320434 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 19:57:55.321042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.321021 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c5ada151-486a-425b-89d5-f21ad262e7f4" containerName="revoke-keys" Oct 30 19:57:55.321152 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.321044 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ada151-486a-425b-89d5-f21ad262e7f4" containerName="revoke-keys" Oct 30 19:57:55.321250 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.321182 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ada151-486a-425b-89d5-f21ad262e7f4" containerName="revoke-keys" Oct 30 19:57:55.324593 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.324570 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.386888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.386860 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f77d05-openshift-default-dockercfg-m2vw9\"" Oct 30 19:57:55.390195 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.390163 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 19:57:55.390779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.390751 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.390882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.390796 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.390882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.390829 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.390989 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.390966 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc7p\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.391076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.391058 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.391158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.391143 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.391260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.391173 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.391260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.391204 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.391260 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.391253 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494293 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494402 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494435 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494471 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494496 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494578 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494617 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494655 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.494698 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc7p\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.495836 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.496439 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.496137 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.498069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.497725 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.498069 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.497818 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.498294 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.498068 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.500461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.500431 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.500596 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.500509 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.536329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.536304 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" Oct 30 19:57:55.536485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.536357 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp" event={"ID":"c5ada151-486a-425b-89d5-f21ad262e7f4","Type":"ContainerDied","Data":"dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666"} Oct 30 19:57:55.536485 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.536440 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd0ab7750068c61c84295639c9e26ae3b9cfa6dd07c914320c82a45a051f3666" Oct 30 19:57:55.559118 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.559092 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc7p\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.575103 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.575051 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\") pod \"e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:55.639172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:55.639139 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:56.073330 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.073302 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:57:56.080053 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.080030 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 19:57:56.083480 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:57:56.083406 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8e50bf_801f_4219_b2db_93f41a3c4784.slice/crio-b5032e650a68cf10af7d33de9864d92c73248c320c0e695eda26790bf386b807 WatchSource:0}: Error finding container b5032e650a68cf10af7d33de9864d92c73248c320c0e695eda26790bf386b807: Status 404 returned error can't find the container with id b5032e650a68cf10af7d33de9864d92c73248c320c0e695eda26790bf386b807 Oct 30 19:57:56.086165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.086137 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:56.086283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.086198 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:56.086283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.086256 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:57:56.206574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206535 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206633 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206680 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206712 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206747 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206742 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206826 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206870 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.206993 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206948 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.207169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.206999 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\") pod \"2c57b60a-be79-46a7-999b-639942a1c038\" (UID: \"2c57b60a-be79-46a7-999b-639942a1c038\") " Oct 30 19:57:56.209313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.209283 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:56.210288 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.209996 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:56.210649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.210623 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:56.210951 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.210929 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:56.211758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.211727 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "kube-api-access-6pgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:56.212192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.212165 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:56.212523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.212498 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:56.213474 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.213450 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:56.213650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.213630 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo" pod "2c57b60a-be79-46a7-999b-639942a1c038" (UID: "2c57b60a-be79-46a7-999b-639942a1c038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:56.308499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308460 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308491 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pgvc\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-kube-api-access-6pgvc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308499 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308506 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57b60a-be79-46a7-999b-639942a1c038-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308519 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308528 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308536 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57b60a-be79-46a7-999b-639942a1c038-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308544 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308552 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57b60a-be79-46a7-999b-639942a1c038-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.308744 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.308560 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57b60a-be79-46a7-999b-639942a1c038-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:56.546308 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.546264 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerStarted","Data":"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd"} Oct 30 19:57:56.546521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.546327 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerStarted","Data":"b5032e650a68cf10af7d33de9864d92c73248c320c0e695eda26790bf386b807"} Oct 30 19:57:56.548278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.548252 4789 generic.go:356] "Generic (PLEG): container finished" podID="2c57b60a-be79-46a7-999b-639942a1c038" containerID="748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f" exitCode=0 Oct 30 19:57:56.548407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.548297 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" Oct 30 19:57:56.548407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.548334 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerDied","Data":"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f"} Oct 30 19:57:56.548407 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.548394 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8" event={"ID":"2c57b60a-be79-46a7-999b-639942a1c038","Type":"ContainerDied","Data":"3bc242d52077b67f8b865f1f72fc4535c013bff02219704111b7a6d807a2cfdf"} Oct 30 19:57:56.548523 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.548427 4789 scope.go:122] "RemoveContainer" containerID="748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f" Oct 30 19:57:56.562080 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.562050 4789 scope.go:122] "RemoveContainer" containerID="b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44" Oct 30 19:57:56.575241 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.575092 4789 scope.go:122] "RemoveContainer" containerID="748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f" Oct 30 19:57:56.575488 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:56.575465 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f\": container with ID starting with 748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f not found: ID does not exist" containerID="748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f" Oct 30 19:57:56.575559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.575496 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f"} err="failed to get container status \"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f\": rpc error: code = NotFound desc = could not find container \"748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f\": container with ID starting with 748170d3f384369ea59ac912efbbbcfcc524114a21a56cf2438522f2fe51865f not found: ID does not exist" Oct 30 19:57:56.575559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.575515 4789 scope.go:122] "RemoveContainer" containerID="b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44" Oct 30 19:57:56.575813 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:57:56.575791 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44\": container with ID starting with b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44 not found: ID does not exist" containerID="b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44" Oct 30 19:57:56.575896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.575816 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44"} err="failed to get container status \"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44\": rpc error: code = NotFound desc = could not find container \"b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44\": container with ID starting with b52e8ae357af9cd5d594cfc57a7f321f026f5a5b9df60e65822aa854722bca44 not found: ID does not exist" Oct 30 19:57:56.639414 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.639387 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:56.663363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.663312 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podStartSLOduration=1.663298967 podStartE2EDuration="1.663298967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:56.661985885 +0000 UTC m=+4313.163896370" watchObservedRunningTime="2026-10-30 19:57:56.663298967 +0000 UTC m=+4313.165209452" Oct 30 19:57:56.774888 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.774856 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:57:56.796463 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:56.796365 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe7634-openshift-default-7795f5cc55-647j8"] Oct 30 19:57:57.645587 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:57.645558 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:57.669627 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:57.669592 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c57b60a-be79-46a7-999b-639942a1c038" path="/var/lib/kubelet/pods/2c57b60a-be79-46a7-999b-639942a1c038/volumes" Oct 30 19:57:58.562541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:58.562510 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:57:58.563497 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:57:58.563471 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:58:00.060987 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:00.060927 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" containerID="cri-o://8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148" gracePeriod=25 Oct 30 19:58:04.524928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:04.524889 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w"] Oct 30 19:58:04.566243 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:04.566202 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b2604c-smh9w"] Oct 30 19:58:05.667985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:05.667951 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc721f35-c500-4d52-aa3d-49ebca41a9fb" path="/var/lib/kubelet/pods/bc721f35-c500-4d52-aa3d-49ebca41a9fb/volumes" Oct 30 19:58:05.798179 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:05.798148 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9"] Oct 30 19:58:05.843757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:05.843727 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-312161-pxdh9"] Oct 30 19:58:06.086455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:06.086414 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:06.263020 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:06.262987 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:58:06.364720 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:06.364648 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:58:07.667890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:07.667856 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32709e7-0d47-4789-8f17-9e6c732fe370" path="/var/lib/kubelet/pods/d32709e7-0d47-4789-8f17-9e6c732fe370/volumes" Oct 30 19:58:11.263730 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:11.263669 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" containerID="cri-o://ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048" gracePeriod=25 Oct 30 19:58:11.365530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:11.365470 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" containerID="cri-o://d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061" gracePeriod=25 Oct 30 19:58:12.016490 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:12.016456 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp"] Oct 30 19:58:12.064224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:12.064181 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-59594f-5fknp"] Oct 30 19:58:12.408455 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:12.408356 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:58:12.408926 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:12.408639 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" containerID="cri-o://35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f" gracePeriod=30 Oct 30 19:58:13.442920 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.442883 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr"] Oct 30 19:58:13.443608 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443587 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.443715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443608 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.443715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443633 4789 cpu_manager.go:414] "containerMap: removing container" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.443715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443641 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.443892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443818 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.443892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.443834 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c57b60a-be79-46a7-999b-639942a1c038" containerName="istio-proxy" Oct 30 19:58:13.447184 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.447161 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.477025 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.477001 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.477135 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.477051 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.477190 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.477163 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkr6m\" (UniqueName: \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.497705 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.497684 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:13.497822 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.497701 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:13.543948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.543927 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:58:13.544062 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.543987 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:13.546557 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.546540 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr"] Oct 30 19:58:13.578668 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.578642 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.578785 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.578722 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr6m\" (UniqueName: \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.578859 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.578815 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.579295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.579276 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.581147 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.581129 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.636605 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.636580 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr6m\" (UniqueName: \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\") pod \"maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:13.667392 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.667368 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ada151-486a-425b-89d5-f21ad262e7f4" path="/var/lib/kubelet/pods/c5ada151-486a-425b-89d5-f21ad262e7f4/volumes" Oct 30 19:58:13.761076 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:13.761028 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:14.138628 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:14.138602 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr"] Oct 30 19:58:14.140804 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:14.140774 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73b5b5e_e056_4973_8aa4_31656aa11983.slice/crio-89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186 WatchSource:0}: Error finding container 89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186: Status 404 returned error can't find the container with id 89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186 Oct 30 19:58:14.668382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:14.668308 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" event={"ID":"d73b5b5e-e056-4973-8aa4-31656aa11983","Type":"ContainerStarted","Data":"c523ec39791a270cba2332c23e1152a08490ef3e3e8593aeca18d52ea0cf725c"} Oct 30 19:58:14.668382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:14.668355 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" event={"ID":"d73b5b5e-e056-4973-8aa4-31656aa11983","Type":"ContainerStarted","Data":"89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186"} Oct 30 19:58:14.780224 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:14.780154 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" podStartSLOduration=1.7801354109999998 podStartE2EDuration="1.780135411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:14.778603188 +0000 UTC m=+4331.280513673" watchObservedRunningTime="2026-10-30 19:58:14.780135411 +0000 UTC m=+4331.282045898" Oct 30 19:58:15.675130 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.675092 4789 generic.go:356] "Generic (PLEG): container finished" podID="d73b5b5e-e056-4973-8aa4-31656aa11983" containerID="c523ec39791a270cba2332c23e1152a08490ef3e3e8593aeca18d52ea0cf725c" exitCode=0 Oct 30 19:58:15.675571 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.675167 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" event={"ID":"d73b5b5e-e056-4973-8aa4-31656aa11983","Type":"ContainerDied","Data":"c523ec39791a270cba2332c23e1152a08490ef3e3e8593aeca18d52ea0cf725c"} Oct 30 19:58:15.773111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.773074 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr"] Oct 30 19:58:15.776889 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.776869 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.806632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.806610 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.806762 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.806656 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.806818 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.806749 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwwk\" (UniqueName: \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.835810 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.835780 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr"] Oct 30 19:58:15.908325 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.908291 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.908512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.908356 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.908512 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.908413 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwwk\" (UniqueName: \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.909201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.909170 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.910641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.910613 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:15.966621 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:15.966591 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwwk\" (UniqueName: \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:16.085895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.085858 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:16.089954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.089926 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:16.476398 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.476367 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr"] Oct 30 19:58:16.478638 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:16.478605 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecd110b_dd2c_41af_bbd5_52f099ec3743.slice/crio-14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b WatchSource:0}: Error finding container 14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b: Status 404 returned error can't find the container with id 14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b Oct 30 19:58:16.683802 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.683767 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" event={"ID":"0ecd110b-dd2c-41af-bbd5-52f099ec3743","Type":"ContainerStarted","Data":"14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b"} Oct 30 19:58:16.826958 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.826928 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:16.921124 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.921099 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\") pod \"d73b5b5e-e056-4973-8aa4-31656aa11983\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " Oct 30 19:58:16.921281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.921166 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\") pod \"d73b5b5e-e056-4973-8aa4-31656aa11983\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " Oct 30 19:58:16.921281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.921270 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\" (UniqueName: \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\") pod \"d73b5b5e-e056-4973-8aa4-31656aa11983\" (UID: \"d73b5b5e-e056-4973-8aa4-31656aa11983\") " Oct 30 19:58:16.921758 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.921731 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca" pod "d73b5b5e-e056-4973-8aa4-31656aa11983" (UID: "d73b5b5e-e056-4973-8aa4-31656aa11983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:16.923815 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.923785 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca" pod "d73b5b5e-e056-4973-8aa4-31656aa11983" (UID: "d73b5b5e-e056-4973-8aa4-31656aa11983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:16.923906 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:16.923881 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m" pod "d73b5b5e-e056-4973-8aa4-31656aa11983" (UID: "d73b5b5e-e056-4973-8aa4-31656aa11983"). InnerVolumeSpecName "kube-api-access-dkr6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:17.023337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.023262 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.023337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.023292 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73b5b5e-e056-4973-8aa4-31656aa11983-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.023337 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.023304 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkr6m\" (UniqueName: \"kubernetes.io/projected/d73b5b5e-e056-4973-8aa4-31656aa11983-kube-api-access-dkr6m\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.668851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.668754 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:17.669299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.669264 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:17.685159 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.685136 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:58:17.690611 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.690587 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" Oct 30 19:58:17.690692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.690635 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr" event={"ID":"d73b5b5e-e056-4973-8aa4-31656aa11983","Type":"ContainerDied","Data":"89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186"} Oct 30 19:58:17.690692 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.690678 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89b18e46c7777193efbc85a51b514608c8d61fb3c28da7b75ccaf94b82fea186" Oct 30 19:58:17.693555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.693529 4789 generic.go:356] "Generic (PLEG): container finished" podID="0ecd110b-dd2c-41af-bbd5-52f099ec3743" containerID="1829a9c53734eee23098d83d79db76a27555f09690a8c5e01c76d0a34544fab5" exitCode=0 Oct 30 19:58:17.693640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.693602 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" event={"ID":"0ecd110b-dd2c-41af-bbd5-52f099ec3743","Type":"ContainerDied","Data":"1829a9c53734eee23098d83d79db76a27555f09690a8c5e01c76d0a34544fab5"} Oct 30 19:58:17.695261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.695237 4789 generic.go:356] "Generic (PLEG): container finished" podID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerID="35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f" exitCode=0 Oct 30 19:58:17.695372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.695280 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerDied","Data":"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f"} Oct 30 19:58:17.695372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.695287 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" Oct 30 19:58:17.695372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.695323 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x" event={"ID":"cc5a5f33-6e4c-4238-823e-feb61db1cdd3","Type":"ContainerDied","Data":"fd68cf0a76aad0425938b0dbb50e25c7c2aa75fb22a2d1da6a34036ac224a3e2"} Oct 30 19:58:17.695372 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.695355 4789 scope.go:122] "RemoveContainer" containerID="35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f" Oct 30 19:58:17.705413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.705380 4789 scope.go:122] "RemoveContainer" containerID="9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907" Oct 30 19:58:17.714448 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.714423 4789 scope.go:122] "RemoveContainer" containerID="35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f" Oct 30 19:58:17.714742 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:17.714721 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f\": container with ID starting with 35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f not found: ID does not exist" containerID="35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f" Oct 30 19:58:17.714832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.714752 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f"} err="failed to get container status \"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f\": rpc error: code = NotFound desc = could not find container \"35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f\": container with ID starting with 35b36c8e64e787d13ac019cda2e2a99396a7df6d747ffda08af8efe44b4f446f not found: ID does not exist" Oct 30 19:58:17.714832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.714775 4789 scope.go:122] "RemoveContainer" containerID="9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907" Oct 30 19:58:17.715068 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:17.715050 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907\": container with ID starting with 9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907 not found: ID does not exist" containerID="9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907" Oct 30 19:58:17.715111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.715081 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907"} err="failed to get container status \"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907\": rpc error: code = NotFound desc = could not find container \"9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907\": container with ID starting with 9e5879cfa86eac1f1eabb35bf14c5b36a42729cf74d75d82885dced721aeb907 not found: ID does not exist" Oct 30 19:58:17.730907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.730882 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.730921 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.730975 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731018 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731181 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731152 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:17.731258 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731224 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731321 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731270 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731339 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731377 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731419 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\") pod \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\" (UID: \"cc5a5f33-6e4c-4238-823e-feb61db1cdd3\") " Oct 30 19:58:17.731574 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731501 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:17.731737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731715 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:17.731914 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731891 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.731983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731922 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.731983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731938 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.731983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731943 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:17.731983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.731655 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:17.733493 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.733459 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:17.733918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.733892 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "kube-api-access-z6vm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:17.733918 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.733905 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:17.734060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.734030 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo" pod "cc5a5f33-6e4c-4238-823e-feb61db1cdd3" (UID: "cc5a5f33-6e4c-4238-823e-feb61db1cdd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:17.832759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832735 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.832759 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832758 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.832956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832767 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.832956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832778 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vm5\" (UniqueName: \"kubernetes.io/projected/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-kube-api-access-z6vm5\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.832956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832787 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.832956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.832796 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5a5f33-6e4c-4238-823e-feb61db1cdd3-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:17.985466 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:17.985432 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:58:18.034630 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.034582 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:58:18.197425 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:18.197391 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:58:18.332949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.332912 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:58:18.337205 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.337174 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b2604c-openshift-default-bb6d5b8d5-4vq7x"] Oct 30 19:58:18.843363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.843339 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:18.942936 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.942906 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\" (UniqueName: \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\") pod \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " Oct 30 19:58:18.943089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.943074 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\") pod \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " Oct 30 19:58:18.943138 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.943102 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\") pod \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\" (UID: \"0ecd110b-dd2c-41af-bbd5-52f099ec3743\") " Oct 30 19:58:18.943473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.943441 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca" pod "0ecd110b-dd2c-41af-bbd5-52f099ec3743" (UID: "0ecd110b-dd2c-41af-bbd5-52f099ec3743"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:18.944974 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.944954 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk" pod "0ecd110b-dd2c-41af-bbd5-52f099ec3743" (UID: "0ecd110b-dd2c-41af-bbd5-52f099ec3743"). InnerVolumeSpecName "kube-api-access-dgwwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:18.945162 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:18.945141 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca" pod "0ecd110b-dd2c-41af-bbd5-52f099ec3743" (UID: "0ecd110b-dd2c-41af-bbd5-52f099ec3743"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:19.045187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.045111 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:19.045187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.045146 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecd110b-dd2c-41af-bbd5-52f099ec3743-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:19.045187 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.045163 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwwk\" (UniqueName: \"kubernetes.io/projected/0ecd110b-dd2c-41af-bbd5-52f099ec3743-kube-api-access-dgwwk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:19.669295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.669261 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" path="/var/lib/kubelet/pods/cc5a5f33-6e4c-4238-823e-feb61db1cdd3/volumes" Oct 30 19:58:19.710469 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.710446 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" Oct 30 19:58:19.710589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.710481 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr" event={"ID":"0ecd110b-dd2c-41af-bbd5-52f099ec3743","Type":"ContainerDied","Data":"14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b"} Oct 30 19:58:19.710589 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:19.710532 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f06a40bbcf748d10fa91a3db2fe8554ab2febe8d08a14ead95032bc9e73f5b" Oct 30 19:58:20.359024 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.358996 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:58:20.457421 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.457390 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\") pod \"dbd3ef15-5711-43f4-9141-8b266ed683fb\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " Oct 30 19:58:20.457624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.457474 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\" (UniqueName: \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\") pod \"dbd3ef15-5711-43f4-9141-8b266ed683fb\" (UID: \"dbd3ef15-5711-43f4-9141-8b266ed683fb\") " Oct 30 19:58:20.457825 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.457800 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume" pod "dbd3ef15-5711-43f4-9141-8b266ed683fb" (UID: "dbd3ef15-5711-43f4-9141-8b266ed683fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:20.459618 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.459595 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r" pod "dbd3ef15-5711-43f4-9141-8b266ed683fb" (UID: "dbd3ef15-5711-43f4-9141-8b266ed683fb"). InnerVolumeSpecName "kube-api-access-t9c8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:20.559078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.559045 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9c8r\" (UniqueName: \"kubernetes.io/projected/dbd3ef15-5711-43f4-9141-8b266ed683fb-kube-api-access-t9c8r\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:20.559078 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.559073 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd3ef15-5711-43f4-9141-8b266ed683fb-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:20.717907 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.717873 4789 generic.go:356] "Generic (PLEG): container finished" podID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerID="8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148" exitCode=0 Oct 30 19:58:20.718106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.717927 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" Oct 30 19:58:20.718106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.717956 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" event={"ID":"dbd3ef15-5711-43f4-9141-8b266ed683fb","Type":"ContainerDied","Data":"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148"} Oct 30 19:58:20.718106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.718008 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8" event={"ID":"dbd3ef15-5711-43f4-9141-8b266ed683fb","Type":"ContainerDied","Data":"05486ef20cf2eb4d4b88d9781762c7b0ba053938b9acff36f47fb8c8a0c41313"} Oct 30 19:58:20.718106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.718035 4789 scope.go:122] "RemoveContainer" containerID="8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148" Oct 30 19:58:20.728551 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.728535 4789 scope.go:122] "RemoveContainer" containerID="8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148" Oct 30 19:58:20.728789 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:20.728771 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148\": container with ID starting with 8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148 not found: ID does not exist" containerID="8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148" Oct 30 19:58:20.728839 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.728795 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148"} err="failed to get container status \"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148\": rpc error: code = NotFound desc = could not find container \"8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148\": container with ID starting with 8e75bc9f62e79333936cd9727a2e74b16efea1a9f4160a58fc48764c21d04148 not found: ID does not exist" Oct 30 19:58:20.874164 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.874136 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:58:20.892217 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:20.892190 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b2604c-79f7bfdcff-wg7t8"] Oct 30 19:58:21.319569 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.319532 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5"] Oct 30 19:58:21.320033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320016 4789 cpu_manager.go:414] "containerMap: removing container" podUID="0ecd110b-dd2c-41af-bbd5-52f099ec3743" containerName="revoke-keys" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320036 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecd110b-dd2c-41af-bbd5-52f099ec3743" containerName="revoke-keys" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320048 4789 cpu_manager.go:414] "containerMap: removing container" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320053 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320066 4789 cpu_manager.go:414] "containerMap: removing container" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320073 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320083 4789 cpu_manager.go:414] "containerMap: removing container" podUID="d73b5b5e-e056-4973-8aa4-31656aa11983" containerName="revoke-keys" Oct 30 19:58:21.320093 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320088 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73b5b5e-e056-4973-8aa4-31656aa11983" containerName="revoke-keys" Oct 30 19:58:21.320346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320186 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecd110b-dd2c-41af-bbd5-52f099ec3743" containerName="revoke-keys" Oct 30 19:58:21.320346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320196 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:21.320346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320205 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" containerName="payload-pre-processing" Oct 30 19:58:21.320346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.320222 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73b5b5e-e056-4973-8aa4-31656aa11983" containerName="revoke-keys" Oct 30 19:58:21.324632 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.324614 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.367030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.367007 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.367367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.367063 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27zq\" (UniqueName: \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.367367 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.367170 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.375660 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.375631 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:58:21.376126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.376102 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:21.376261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.376130 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:21.376363 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.376342 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:21.379257 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.379233 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5"] Oct 30 19:58:21.468156 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.468128 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.468329 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.468235 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.468394 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.468333 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27zq\" (UniqueName: \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.468837 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.468814 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.470534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.470510 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.525566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.525544 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27zq\" (UniqueName: \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\") pod \"maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.637798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.637749 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:21.668879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:21.668844 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd3ef15-5711-43f4-9141-8b266ed683fb" path="/var/lib/kubelet/pods/dbd3ef15-5711-43f4-9141-8b266ed683fb/volumes" Oct 30 19:58:22.008880 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:22.008857 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5"] Oct 30 19:58:22.011241 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:22.011199 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6922fa7_e81d_4ea1_9384_12308de983de.slice/crio-a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6 WatchSource:0}: Error finding container a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6: Status 404 returned error can't find the container with id a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6 Oct 30 19:58:22.740292 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:22.740252 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" event={"ID":"a6922fa7-e81d-4ea1-9384-12308de983de","Type":"ContainerStarted","Data":"d837a8f82fc679328218ef88843ecf2342882bb5e018a7897c7bd5131b71ae4b"} Oct 30 19:58:22.740751 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:22.740323 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" event={"ID":"a6922fa7-e81d-4ea1-9384-12308de983de","Type":"ContainerStarted","Data":"a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6"} Oct 30 19:58:22.856166 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:22.856120 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" podStartSLOduration=1.8561052839999999 podStartE2EDuration="1.856105284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:22.852716287 +0000 UTC m=+4339.354626775" watchObservedRunningTime="2026-10-30 19:58:22.856105284 +0000 UTC m=+4339.358015769" Oct 30 19:58:23.748032 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:23.747994 4789 generic.go:356] "Generic (PLEG): container finished" podID="a6922fa7-e81d-4ea1-9384-12308de983de" containerID="d837a8f82fc679328218ef88843ecf2342882bb5e018a7897c7bd5131b71ae4b" exitCode=0 Oct 30 19:58:23.748484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:23.748056 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" event={"ID":"a6922fa7-e81d-4ea1-9384-12308de983de","Type":"ContainerDied","Data":"d837a8f82fc679328218ef88843ecf2342882bb5e018a7897c7bd5131b71ae4b"} Oct 30 19:58:24.895511 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:24.895487 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:25.001582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.001556 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\") pod \"a6922fa7-e81d-4ea1-9384-12308de983de\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " Oct 30 19:58:25.001691 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.001633 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\" (UniqueName: \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\") pod \"a6922fa7-e81d-4ea1-9384-12308de983de\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " Oct 30 19:58:25.001813 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.001791 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\") pod \"a6922fa7-e81d-4ea1-9384-12308de983de\" (UID: \"a6922fa7-e81d-4ea1-9384-12308de983de\") " Oct 30 19:58:25.002082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.002053 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca" pod "a6922fa7-e81d-4ea1-9384-12308de983de" (UID: "a6922fa7-e81d-4ea1-9384-12308de983de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:25.002233 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.002198 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:25.003639 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.003616 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca" pod "a6922fa7-e81d-4ea1-9384-12308de983de" (UID: "a6922fa7-e81d-4ea1-9384-12308de983de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:25.003743 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.003714 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq" pod "a6922fa7-e81d-4ea1-9384-12308de983de" (UID: "a6922fa7-e81d-4ea1-9384-12308de983de"). InnerVolumeSpecName "kube-api-access-r27zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:25.103316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.103285 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6922fa7-e81d-4ea1-9384-12308de983de-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:25.103316 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.103315 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27zq\" (UniqueName: \"kubernetes.io/projected/a6922fa7-e81d-4ea1-9384-12308de983de-kube-api-access-r27zq\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:25.762484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.762456 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" Oct 30 19:58:25.762624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.762483 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5" event={"ID":"a6922fa7-e81d-4ea1-9384-12308de983de","Type":"ContainerDied","Data":"a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6"} Oct 30 19:58:25.762624 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.762521 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5294324e34f7258337242d594447b168ee79dc02558b263c908f19120c58dd6" Oct 30 19:58:25.829317 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.829285 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:58:25.867278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:25.867253 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:58:27.673391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:27.673351 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:27.673878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:27.673794 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:58:29.163402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:29.163369 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx"] Oct 30 19:58:29.209798 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:29.209770 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-d6f58f-rpvlx"] Oct 30 19:58:29.667829 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:29.667799 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3b2342-3a74-42dc-b63f-50dd13777d73" path="/var/lib/kubelet/pods/df3b2342-3a74-42dc-b63f-50dd13777d73/volumes" Oct 30 19:58:31.541468 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.541431 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:58:31.636503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.636477 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:58:31.677058 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.677029 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\" (UniqueName: \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\") pod \"c80c1d37-3d0a-457a-93cc-ff28894122b4\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " Oct 30 19:58:31.677220 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.677082 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\") pod \"c80c1d37-3d0a-457a-93cc-ff28894122b4\" (UID: \"c80c1d37-3d0a-457a-93cc-ff28894122b4\") " Oct 30 19:58:31.677475 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.677449 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume" pod "c80c1d37-3d0a-457a-93cc-ff28894122b4" (UID: "c80c1d37-3d0a-457a-93cc-ff28894122b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:31.679251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.679225 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9" pod "c80c1d37-3d0a-457a-93cc-ff28894122b4" (UID: "c80c1d37-3d0a-457a-93cc-ff28894122b4"). InnerVolumeSpecName "kube-api-access-gm2k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:31.778757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.778696 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\" (UniqueName: \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\") pod \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " Oct 30 19:58:31.778757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.778734 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\") pod \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\" (UID: \"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f\") " Oct 30 19:58:31.779060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.779039 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm2k9\" (UniqueName: \"kubernetes.io/projected/c80c1d37-3d0a-457a-93cc-ff28894122b4-kube-api-access-gm2k9\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:31.779128 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.779067 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c80c1d37-3d0a-457a-93cc-ff28894122b4-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:31.779169 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.779136 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume" pod "73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" (UID: "73c5cc5f-a63f-4ce6-88d5-ee3236cc546f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:31.781009 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.780983 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv" pod "73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" (UID: "73c5cc5f-a63f-4ce6-88d5-ee3236cc546f"). InnerVolumeSpecName "kube-api-access-swztv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:31.813590 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.813560 4789 generic.go:356] "Generic (PLEG): container finished" podID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerID="d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061" exitCode=0 Oct 30 19:58:31.813725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.813610 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" Oct 30 19:58:31.813725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.813649 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" event={"ID":"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f","Type":"ContainerDied","Data":"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061"} Oct 30 19:58:31.813725 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.813713 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg" event={"ID":"73c5cc5f-a63f-4ce6-88d5-ee3236cc546f","Type":"ContainerDied","Data":"38d86bd22c94440c299608d1f11848d0ef9d40c9350b36682fe9e1af29c618d5"} Oct 30 19:58:31.813913 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.813744 4789 scope.go:122] "RemoveContainer" containerID="d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061" Oct 30 19:58:31.815819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.815787 4789 generic.go:356] "Generic (PLEG): container finished" podID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerID="ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048" exitCode=0 Oct 30 19:58:31.815928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.815821 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" event={"ID":"c80c1d37-3d0a-457a-93cc-ff28894122b4","Type":"ContainerDied","Data":"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048"} Oct 30 19:58:31.815928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.815858 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" event={"ID":"c80c1d37-3d0a-457a-93cc-ff28894122b4","Type":"ContainerDied","Data":"0f362a91b2b4126113dcd5880a3ff56560aeb2d6be1de4e0a9d3a0505878274e"} Oct 30 19:58:31.815928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.815923 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8" Oct 30 19:58:31.826018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.825987 4789 scope.go:122] "RemoveContainer" containerID="d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061" Oct 30 19:58:31.826285 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:31.826265 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061\": container with ID starting with d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061 not found: ID does not exist" containerID="d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061" Oct 30 19:58:31.826383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.826295 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061"} err="failed to get container status \"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061\": rpc error: code = NotFound desc = could not find container \"d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061\": container with ID starting with d12ba78814af1f86c209d76c3602868fde9db457ef3adb719c515260d6c28061 not found: ID does not exist" Oct 30 19:58:31.826383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.826320 4789 scope.go:122] "RemoveContainer" containerID="ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048" Oct 30 19:58:31.835306 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.835288 4789 scope.go:122] "RemoveContainer" containerID="ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048" Oct 30 19:58:31.835620 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:31.835601 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048\": container with ID starting with ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048 not found: ID does not exist" containerID="ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048" Oct 30 19:58:31.835673 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.835627 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048"} err="failed to get container status \"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048\": rpc error: code = NotFound desc = could not find container \"ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048\": container with ID starting with ddf62afc471e11cd90a4dcafec10ae53acb38eb6f7c504cec8561769d609a048 not found: ID does not exist" Oct 30 19:58:31.880304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.880281 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swztv\" (UniqueName: \"kubernetes.io/projected/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-kube-api-access-swztv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:31.880304 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.880301 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:31.975956 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.975936 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:58:31.990809 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:31.990768 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-59594f-757fdf9779-nbhwg"] Oct 30 19:58:32.098315 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:32.098264 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:58:32.105318 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:32.105297 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-59594f-d8d88f5d4-hg7f8"] Oct 30 19:58:33.005895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.005857 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" containerID="cri-o://a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c" gracePeriod=45 Oct 30 19:58:33.055399 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.055361 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" containerID="cri-o://2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd" gracePeriod=45 Oct 30 19:58:33.366994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.366966 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:58:33.369963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.369946 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:58:33.494383 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494350 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\" (UniqueName: \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.494567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494407 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\" (UniqueName: \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.494567 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494439 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.494845 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494821 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.494949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494878 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.494949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494923 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\" (UniqueName: \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.495071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.494975 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.495071 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495032 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.495192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495080 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.495192 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495159 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.495345 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495317 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.495401 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495314 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.495470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495409 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\") pod \"58e67e16-4fe3-4732-9da8-da86d4f1af92\" (UID: \"58e67e16-4fe3-4732-9da8-da86d4f1af92\") " Oct 30 19:58:33.495470 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495446 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\") pod \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\" (UID: \"31ca16be-6cf0-4f92-af5c-4b515d7e7f27\") " Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495847 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.495858 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496049 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496128 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496149 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496164 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496179 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.496312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.496272 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.498346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.498304 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:33.498346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.498321 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.502331 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.502295 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:33.502435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.502378 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:33.503443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.503418 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn" pod "58e67e16-4fe3-4732-9da8-da86d4f1af92" (UID: "58e67e16-4fe3-4732-9da8-da86d4f1af92"). InnerVolumeSpecName "kube-api-access-cj8rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:33.505051 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.505025 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864" pod "31ca16be-6cf0-4f92-af5c-4b515d7e7f27" (UID: "31ca16be-6cf0-4f92-af5c-4b515d7e7f27"). InnerVolumeSpecName "kube-api-access-qd864". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:33.597500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597470 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597500 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597497 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8rn\" (UniqueName: \"kubernetes.io/projected/58e67e16-4fe3-4732-9da8-da86d4f1af92-kube-api-access-cj8rn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597509 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd864\" (UniqueName: \"kubernetes.io/projected/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-kube-api-access-qd864\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597520 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597534 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e67e16-4fe3-4732-9da8-da86d4f1af92-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597543 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e67e16-4fe3-4732-9da8-da86d4f1af92-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597552 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.597729 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.597560 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ca16be-6cf0-4f92-af5c-4b515d7e7f27-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:33.667619 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.667584 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" path="/var/lib/kubelet/pods/73c5cc5f-a63f-4ce6-88d5-ee3236cc546f/volumes" Oct 30 19:58:33.668074 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.668051 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" path="/var/lib/kubelet/pods/c80c1d37-3d0a-457a-93cc-ff28894122b4/volumes" Oct 30 19:58:33.833766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.833701 4789 generic.go:356] "Generic (PLEG): container finished" podID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerID="2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd" exitCode=0 Oct 30 19:58:33.833766 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.833751 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" Oct 30 19:58:33.833941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.833791 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" event={"ID":"31ca16be-6cf0-4f92-af5c-4b515d7e7f27","Type":"ContainerDied","Data":"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd"} Oct 30 19:58:33.833941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.833832 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r" event={"ID":"31ca16be-6cf0-4f92-af5c-4b515d7e7f27","Type":"ContainerDied","Data":"fe2146e6d092fa34a492a6e5b1f35a02d393e07e4cd596d3166c89af26a03e26"} Oct 30 19:58:33.833941 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.833857 4789 scope.go:122] "RemoveContainer" containerID="2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd" Oct 30 19:58:33.835667 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.835592 4789 generic.go:356] "Generic (PLEG): container finished" podID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerID="a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c" exitCode=0 Oct 30 19:58:33.835804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.835703 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" event={"ID":"58e67e16-4fe3-4732-9da8-da86d4f1af92","Type":"ContainerDied","Data":"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c"} Oct 30 19:58:33.835804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.835730 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" event={"ID":"58e67e16-4fe3-4732-9da8-da86d4f1af92","Type":"ContainerDied","Data":"361ef8c06199232a1381fb6b4347a28508c3afa7aef143357c9fbca3085dceff"} Oct 30 19:58:33.835804 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.835786 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq" Oct 30 19:58:33.844090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.844073 4789 scope.go:122] "RemoveContainer" containerID="2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd" Oct 30 19:58:33.844384 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:33.844363 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd\": container with ID starting with 2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd not found: ID does not exist" containerID="2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd" Oct 30 19:58:33.844454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.844390 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd"} err="failed to get container status \"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd\": rpc error: code = NotFound desc = could not find container \"2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd\": container with ID starting with 2099644bb669e78f51b3d5283376cf6fdf42f9522f0f5e4e693731987e6fb6dd not found: ID does not exist" Oct 30 19:58:33.844454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.844409 4789 scope.go:122] "RemoveContainer" containerID="a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c" Oct 30 19:58:33.852460 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.852441 4789 scope.go:122] "RemoveContainer" containerID="a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c" Oct 30 19:58:33.852712 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:33.852690 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c\": container with ID starting with a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c not found: ID does not exist" containerID="a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c" Oct 30 19:58:33.852817 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.852732 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c"} err="failed to get container status \"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c\": rpc error: code = NotFound desc = could not find container \"a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c\": container with ID starting with a546458f8367c6c5f31219d8d29d68699a7afb589c201f398ec49c9f11be133c not found: ID does not exist" Oct 30 19:58:33.994976 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:33.994949 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:58:34.042083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:34.042054 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/simulator-w3-22fb18-kserve-74f986f68f-c6l5r"] Oct 30 19:58:34.147862 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:34.147770 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:58:34.150194 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:34.150172 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-22fb18/distinct-w3-22fb18-kserve-8654f78695-g9ftq"] Oct 30 19:58:35.667774 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:35.667741 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" path="/var/lib/kubelet/pods/31ca16be-6cf0-4f92-af5c-4b515d7e7f27/volumes" Oct 30 19:58:35.668173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:35.668157 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" path="/var/lib/kubelet/pods/58e67e16-4fe3-4732-9da8-da86d4f1af92/volumes" Oct 30 19:58:37.668177 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668142 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx"] Oct 30 19:58:37.668600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668578 4789 cpu_manager.go:414] "containerMap: removing container" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:37.668600 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668592 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668604 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668610 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668635 4789 cpu_manager.go:414] "containerMap: removing container" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668640 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668648 4789 cpu_manager.go:414] "containerMap: removing container" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668653 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668660 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a6922fa7-e81d-4ea1-9384-12308de983de" containerName="revoke-keys" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668665 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6922fa7-e81d-4ea1-9384-12308de983de" containerName="revoke-keys" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668682 4789 cpu_manager.go:414] "containerMap: removing container" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" Oct 30 19:58:37.668690 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668687 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668768 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ca16be-6cf0-4f92-af5c-4b515d7e7f27" containerName="main" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668776 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c5cc5f-a63f-4ce6-88d5-ee3236cc546f" containerName="payload-pre-processing" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668783 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6922fa7-e81d-4ea1-9384-12308de983de" containerName="revoke-keys" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668791 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e67e16-4fe3-4732-9da8-da86d4f1af92" containerName="main" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668798 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5a5f33-6e4c-4238-823e-feb61db1cdd3" containerName="istio-proxy" Oct 30 19:58:37.669042 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.668803 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80c1d37-3d0a-457a-93cc-ff28894122b4" containerName="payload-processing" Oct 30 19:58:37.673566 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.673547 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.721200 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.721169 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx"] Oct 30 19:58:37.764650 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.764593 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:37.764821 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.764803 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:37.764895 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.764804 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:37.769111 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.769092 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:58:37.838126 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.838100 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.838328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.838137 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkbb\" (UniqueName: \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.838328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.838282 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.939606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.939576 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.939746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.939618 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkbb\" (UniqueName: \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.939746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.939667 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.940382 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.940354 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.942144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.942118 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:37.997509 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:37.997484 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkbb\" (UniqueName: \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\") pod \"maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:38.286707 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:38.286650 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:38.662501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:38.662475 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx"] Oct 30 19:58:38.664502 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:38.664475 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962ef5a2_88fb_4943_9b97_09c8c8865bf0.slice/crio-adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31 WatchSource:0}: Error finding container adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31: Status 404 returned error can't find the container with id adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31 Oct 30 19:58:38.872064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:38.872032 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" event={"ID":"962ef5a2-88fb-4943-9b97-09c8c8865bf0","Type":"ContainerStarted","Data":"adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31"} Oct 30 19:58:39.879440 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:39.879374 4789 generic.go:356] "Generic (PLEG): container finished" podID="962ef5a2-88fb-4943-9b97-09c8c8865bf0" containerID="622e6914b7388e9d6a67b5fe88e5feee17d6f704bab6045d2bdc921714e857bb" exitCode=0 Oct 30 19:58:39.879814 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:39.879466 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" event={"ID":"962ef5a2-88fb-4943-9b97-09c8c8865bf0","Type":"ContainerDied","Data":"622e6914b7388e9d6a67b5fe88e5feee17d6f704bab6045d2bdc921714e857bb"} Oct 30 19:58:40.849451 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:40.849392 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" containerID="cri-o://aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b" gracePeriod=45 Oct 30 19:58:40.886502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:40.886468 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" containerID="cri-o://0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313" gracePeriod=45 Oct 30 19:58:40.890090 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:40.890062 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": EOF" Oct 30 19:58:41.028640 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.028604 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:41.050501 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.050453 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Oct 30 19:58:41.136371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.136348 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:58:41.158429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.158408 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:58:41.169540 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.169517 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\") pod \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " Oct 30 19:58:41.169644 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.169581 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\") pod \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " Oct 30 19:58:41.169713 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.169684 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\" (UniqueName: \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\") pod \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\" (UID: \"962ef5a2-88fb-4943-9b97-09c8c8865bf0\") " Oct 30 19:58:41.169882 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.169854 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca" pod "962ef5a2-88fb-4943-9b97-09c8c8865bf0" (UID: "962ef5a2-88fb-4943-9b97-09c8c8865bf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:41.170054 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.170034 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.171786 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.171761 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca" pod "962ef5a2-88fb-4943-9b97-09c8c8865bf0" (UID: "962ef5a2-88fb-4943-9b97-09c8c8865bf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:41.171879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.171812 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb" pod "962ef5a2-88fb-4943-9b97-09c8c8865bf0" (UID: "962ef5a2-88fb-4943-9b97-09c8c8865bf0"). InnerVolumeSpecName "kube-api-access-zhkbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:41.270855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.270827 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.270886 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.270936 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.270977 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271011 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271068 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271045 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271077 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\" (UniqueName: \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271111 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\" (UniqueName: \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271149 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271223 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271199 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.271328 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271314 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.271698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271329 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\") pod \"8ca60daf-a031-47e5-93a5-57ac0d607a36\" (UID: \"8ca60daf-a031-47e5-93a5-57ac0d607a36\") " Oct 30 19:58:41.271698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271390 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\") pod \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\" (UID: \"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7\") " Oct 30 19:58:41.271698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271469 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.271698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.271676 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.272244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272176 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.272244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272186 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.272244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272189 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.272533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272258 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkbb\" (UniqueName: \"kubernetes.io/projected/962ef5a2-88fb-4943-9b97-09c8c8865bf0-kube-api-access-zhkbb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.272533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272275 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.272533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272290 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.272533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272307 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.272533 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.272324 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/962ef5a2-88fb-4943-9b97-09c8c8865bf0-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.273930 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.273900 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:41.274106 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.274069 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "kube-api-access-8shs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:41.274204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.274173 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.274732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.274587 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:41.274732 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.274700 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm" pod "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" (UID: "e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:41.275114 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.275094 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx" pod "8ca60daf-a031-47e5-93a5-57ac0d607a36" (UID: "8ca60daf-a031-47e5-93a5-57ac0d607a36"). InnerVolumeSpecName "kube-api-access-fwxtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:41.374444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374358 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shs6\" (UniqueName: \"kubernetes.io/projected/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-kube-api-access-8shs6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374396 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxtx\" (UniqueName: \"kubernetes.io/projected/8ca60daf-a031-47e5-93a5-57ac0d607a36-kube-api-access-fwxtx\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374411 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374426 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374444 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374439 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca60daf-a031-47e5-93a5-57ac0d607a36-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374451 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374463 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ca60daf-a031-47e5-93a5-57ac0d607a36-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.374811 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.374477 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:41.892559 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.892525 4789 generic.go:356] "Generic (PLEG): container finished" podID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerID="aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b" exitCode=0 Oct 30 19:58:41.892985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.892584 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" Oct 30 19:58:41.892985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.892651 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" event={"ID":"8ca60daf-a031-47e5-93a5-57ac0d607a36","Type":"ContainerDied","Data":"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b"} Oct 30 19:58:41.892985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.892700 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5" event={"ID":"8ca60daf-a031-47e5-93a5-57ac0d607a36","Type":"ContainerDied","Data":"da2bbbd082358b290510ede2dffda65363f5cbc383b78d1b115631bf587854c2"} Oct 30 19:58:41.892985 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.892733 4789 scope.go:122] "RemoveContainer" containerID="aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b" Oct 30 19:58:41.894352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.894330 4789 generic.go:356] "Generic (PLEG): container finished" podID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerID="0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313" exitCode=0 Oct 30 19:58:41.894484 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.894470 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" Oct 30 19:58:41.895011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.894893 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" event={"ID":"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7","Type":"ContainerDied","Data":"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313"} Oct 30 19:58:41.895011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.894928 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp" event={"ID":"e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7","Type":"ContainerDied","Data":"f843c163b450cb48f99f7d250fd5fc1a051d765f7ba47d939ee5a7e24df1fdbe"} Oct 30 19:58:41.896242 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.896223 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" Oct 30 19:58:41.896925 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.896298 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx" event={"ID":"962ef5a2-88fb-4943-9b97-09c8c8865bf0","Type":"ContainerDied","Data":"adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31"} Oct 30 19:58:41.897018 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.896976 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adff01f361f87b66cc8aaf72ea53f3faa7b812449100411d66eada7ebd133f31" Oct 30 19:58:41.905737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.905721 4789 scope.go:122] "RemoveContainer" containerID="aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b" Oct 30 19:58:41.906006 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:41.905989 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b\": container with ID starting with aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b not found: ID does not exist" containerID="aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b" Oct 30 19:58:41.906060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.906016 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b"} err="failed to get container status \"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b\": rpc error: code = NotFound desc = could not find container \"aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b\": container with ID starting with aebdfe17aa042078d532b307efdb052fce73fdb04acb66379de3a77e948d469b not found: ID does not exist" Oct 30 19:58:41.906060 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.906033 4789 scope.go:122] "RemoveContainer" containerID="0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313" Oct 30 19:58:41.916044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.916027 4789 scope.go:122] "RemoveContainer" containerID="0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313" Oct 30 19:58:41.916338 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:41.916318 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313\": container with ID starting with 0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313 not found: ID does not exist" containerID="0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313" Oct 30 19:58:41.916402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:41.916350 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313"} err="failed to get container status \"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313\": rpc error: code = NotFound desc = could not find container \"0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313\": container with ID starting with 0f49a3ff7bbf906c86204032519bdddd2f1a2d5b385cd5260c7bf9cf7b28b313 not found: ID does not exist" Oct 30 19:58:42.097961 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:42.097928 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:58:42.103172 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:42.103138 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/premium-w1-9275c3-kserve-5899cb8d4b-jr5b5"] Oct 30 19:58:42.220894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:42.220869 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:58:42.232745 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:42.232708 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9275c3/simulator-w1-9275c3-kserve-5587c8d8ff-tsxxp"] Oct 30 19:58:43.667082 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:43.667050 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" path="/var/lib/kubelet/pods/8ca60daf-a031-47e5-93a5-57ac0d607a36/volumes" Oct 30 19:58:43.667521 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:43.667474 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" path="/var/lib/kubelet/pods/e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7/volumes" Oct 30 19:58:48.196734 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:58:48.196678 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:58:48.248171 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248140 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z"] Oct 30 19:58:48.248609 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248595 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" Oct 30 19:58:48.248653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248610 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" Oct 30 19:58:48.248653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248628 4789 cpu_manager.go:414] "containerMap: removing container" podUID="962ef5a2-88fb-4943-9b97-09c8c8865bf0" containerName="revoke-keys" Oct 30 19:58:48.248653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248633 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962ef5a2-88fb-4943-9b97-09c8c8865bf0" containerName="revoke-keys" Oct 30 19:58:48.248653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248643 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" Oct 30 19:58:48.248653 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248649 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" Oct 30 19:58:48.248826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248731 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca60daf-a031-47e5-93a5-57ac0d607a36" containerName="main" Oct 30 19:58:48.248826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248739 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="962ef5a2-88fb-4943-9b97-09c8c8865bf0" containerName="revoke-keys" Oct 30 19:58:48.248826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.248746 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a3b803-e7fc-4322-aaa4-a8cc0e725cf7" containerName="main" Oct 30 19:58:48.253520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.253504 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.313449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.313426 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:48.315896 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.315875 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:48.316610 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.316587 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2"] Oct 30 19:58:48.320142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.320123 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.333603 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333581 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz9r\" (UniqueName: \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.333698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333622 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.333698 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333649 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.333816 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333804 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.333851 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333833 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.333891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.333856 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvx64\" (UniqueName: \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.354693 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.354670 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:48.354885 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.354863 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:58:48.357041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.357018 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z"] Oct 30 19:58:48.383302 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.383283 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2"] Oct 30 19:58:48.434776 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.434754 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz9r\" (UniqueName: \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.434870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.434787 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.434870 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.434812 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.434948 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.434882 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.435107 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.435087 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.435146 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.435125 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvx64\" (UniqueName: \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.435528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.435485 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.435740 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.435722 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.437199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.437177 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.437199 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.437186 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.493049 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.492999 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvx64\" (UniqueName: \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\") pod \"maas-api-revoke-sub-d5cabe1d-mhjv2\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.494283 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.494258 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz9r\" (UniqueName: \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\") pod \"maas-api-revoke-sub-aed74471-qvs6z\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.566077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.566057 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:48.633750 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.633526 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:48.939449 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.939416 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z"] Oct 30 19:58:48.942988 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:48.942953 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bb431c_f448_49a9_9557_0a98da55fbb9.slice/crio-b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd WatchSource:0}: Error finding container b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd: Status 404 returned error can't find the container with id b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd Oct 30 19:58:48.955473 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:48.955443 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" event={"ID":"f6bb431c-f448-49a9-9557-0a98da55fbb9","Type":"ContainerStarted","Data":"b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd"} Oct 30 19:58:49.039101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.039071 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2"] Oct 30 19:58:49.042544 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:49.042511 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d256cb_f374_476d_808a_8d2bb9c8b794.slice/crio-412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab WatchSource:0}: Error finding container 412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab: Status 404 returned error can't find the container with id 412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab Oct 30 19:58:49.963153 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.963128 4789 generic.go:356] "Generic (PLEG): container finished" podID="f6d256cb-f374-476d-808a-8d2bb9c8b794" containerID="ac28b1dc8f71f0f09099063ff0d50cabb24805bd8c143342d3e8a1a8c6c644fb" exitCode=0 Oct 30 19:58:49.963555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.963220 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" event={"ID":"f6d256cb-f374-476d-808a-8d2bb9c8b794","Type":"ContainerDied","Data":"ac28b1dc8f71f0f09099063ff0d50cabb24805bd8c143342d3e8a1a8c6c644fb"} Oct 30 19:58:49.963555 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.963286 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" event={"ID":"f6d256cb-f374-476d-808a-8d2bb9c8b794","Type":"ContainerStarted","Data":"412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab"} Oct 30 19:58:49.964928 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.964906 4789 generic.go:356] "Generic (PLEG): container finished" podID="f6bb431c-f448-49a9-9557-0a98da55fbb9" containerID="86c0e353ba4bcf78f4aa5b742602400ce417d7c1f08f1dd333cbcf476d5cfa41" exitCode=0 Oct 30 19:58:49.965030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:49.964984 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" event={"ID":"f6bb431c-f448-49a9-9557-0a98da55fbb9","Type":"ContainerDied","Data":"86c0e353ba4bcf78f4aa5b742602400ce417d7c1f08f1dd333cbcf476d5cfa41"} Oct 30 19:58:51.160676 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.160653 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:51.184092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.184072 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:51.258805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258742 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\" (UniqueName: \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\") pod \"f6d256cb-f374-476d-808a-8d2bb9c8b794\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " Oct 30 19:58:51.258805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258803 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\") pod \"f6bb431c-f448-49a9-9557-0a98da55fbb9\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " Oct 30 19:58:51.258994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258823 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\" (UniqueName: \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\") pod \"f6bb431c-f448-49a9-9557-0a98da55fbb9\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " Oct 30 19:58:51.258994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258849 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\") pod \"f6d256cb-f374-476d-808a-8d2bb9c8b794\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " Oct 30 19:58:51.258994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258866 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\") pod \"f6d256cb-f374-476d-808a-8d2bb9c8b794\" (UID: \"f6d256cb-f374-476d-808a-8d2bb9c8b794\") " Oct 30 19:58:51.258994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.258909 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\") pod \"f6bb431c-f448-49a9-9557-0a98da55fbb9\" (UID: \"f6bb431c-f448-49a9-9557-0a98da55fbb9\") " Oct 30 19:58:51.259269 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.259238 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca" pod "f6d256cb-f374-476d-808a-8d2bb9c8b794" (UID: "f6d256cb-f374-476d-808a-8d2bb9c8b794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:51.259477 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.259441 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca" pod "f6bb431c-f448-49a9-9557-0a98da55fbb9" (UID: "f6bb431c-f448-49a9-9557-0a98da55fbb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:51.261101 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.261069 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r" pod "f6bb431c-f448-49a9-9557-0a98da55fbb9" (UID: "f6bb431c-f448-49a9-9557-0a98da55fbb9"). InnerVolumeSpecName "kube-api-access-mtz9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:51.261185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.261092 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca" pod "f6bb431c-f448-49a9-9557-0a98da55fbb9" (UID: "f6bb431c-f448-49a9-9557-0a98da55fbb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:51.261185 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.261099 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64" pod "f6d256cb-f374-476d-808a-8d2bb9c8b794" (UID: "f6d256cb-f374-476d-808a-8d2bb9c8b794"). InnerVolumeSpecName "kube-api-access-dvx64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:51.261299 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.261283 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca" pod "f6d256cb-f374-476d-808a-8d2bb9c8b794" (UID: "f6d256cb-f374-476d-808a-8d2bb9c8b794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:58:51.360230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360170 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvx64\" (UniqueName: \"kubernetes.io/projected/f6d256cb-f374-476d-808a-8d2bb9c8b794-kube-api-access-dvx64\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.360230 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360224 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.360441 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360240 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz9r\" (UniqueName: \"kubernetes.io/projected/f6bb431c-f448-49a9-9557-0a98da55fbb9-kube-api-access-mtz9r\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.360441 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360253 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.360441 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360267 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6d256cb-f374-476d-808a-8d2bb9c8b794-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.360441 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.360281 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6bb431c-f448-49a9-9557-0a98da55fbb9-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:51.464454 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.464421 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:58:51.981790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.981755 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" Oct 30 19:58:51.981983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.981790 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2" event={"ID":"f6d256cb-f374-476d-808a-8d2bb9c8b794","Type":"ContainerDied","Data":"412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab"} Oct 30 19:58:51.981983 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.981839 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412a35df08126afe75c2e20129fd83c3e43ac3b22fc3b4c442733f410446edab" Oct 30 19:58:51.983123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.983102 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" Oct 30 19:58:51.983123 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.983110 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z" event={"ID":"f6bb431c-f448-49a9-9557-0a98da55fbb9","Type":"ContainerDied","Data":"b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd"} Oct 30 19:58:51.983341 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:51.983144 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50e9db3a3f0ccf5a024ac5c9cdef7c595d71226ff97c556eac19f35013eb3bd" Oct 30 19:58:56.465435 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:56.465384 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" containerID="cri-o://7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3" gracePeriod=25 Oct 30 19:58:58.099064 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099027 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q"] Oct 30 19:58:58.099715 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099692 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f6bb431c-f448-49a9-9557-0a98da55fbb9" containerName="revoke-keys" Oct 30 19:58:58.099808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099715 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6bb431c-f448-49a9-9557-0a98da55fbb9" containerName="revoke-keys" Oct 30 19:58:58.099808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099760 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f6d256cb-f374-476d-808a-8d2bb9c8b794" containerName="revoke-keys" Oct 30 19:58:58.099808 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099770 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d256cb-f374-476d-808a-8d2bb9c8b794" containerName="revoke-keys" Oct 30 19:58:58.099991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099893 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d256cb-f374-476d-808a-8d2bb9c8b794" containerName="revoke-keys" Oct 30 19:58:58.099991 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.099913 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6bb431c-f448-49a9-9557-0a98da55fbb9" containerName="revoke-keys" Oct 30 19:58:58.104634 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.104610 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.118198 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.118167 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.118335 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.118234 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gw56\" (UniqueName: \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.118396 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.118350 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.156681 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.156648 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8"] Oct 30 19:58:58.161030 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.161007 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.200586 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.200560 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:58.200702 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.200626 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:58:58.203295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.203277 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:58:58.203295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.203279 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:58:58.205790 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.205766 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q"] Oct 30 19:58:58.219015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.218993 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.219108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.219045 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.219108 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.219066 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp72w\" (UniqueName: \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.219251 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.219230 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gw56\" (UniqueName: \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.219352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.219319 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.219537 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.219387 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.220313 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.220289 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.223281 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.223003 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.224092 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.224071 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8"] Oct 30 19:58:58.277891 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.277869 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gw56\" (UniqueName: \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\") pod \"maas-api-revoke-sub-e2ab9f3a-rrp7q\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.320502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.320475 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.320598 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.320561 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp72w\" (UniqueName: \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.320656 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.320639 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.321482 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.321446 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.322757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.322734 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.388635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.388576 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp72w\" (UniqueName: \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\") pod \"maas-api-revoke-sub-5bbaa2a2-jjrz8\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.420503 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.420479 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:58:58.480395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.480372 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:58:58.793404 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.793375 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q"] Oct 30 19:58:58.795370 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:58.795340 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8c9f97_03a7_4f54_a2f4_5e009d16a5fc.slice/crio-06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b WatchSource:0}: Error finding container 06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b: Status 404 returned error can't find the container with id 06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b Oct 30 19:58:58.890582 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:58.890556 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8"] Oct 30 19:58:58.892824 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:58:58.892791 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b4b59f_8ecb_40c3_b4d7_4dcfd13a55f0.slice/crio-f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a WatchSource:0}: Error finding container f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a: Status 404 returned error can't find the container with id f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a Oct 30 19:58:59.053158 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:59.053072 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" event={"ID":"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0","Type":"ContainerStarted","Data":"f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a"} Oct 30 19:58:59.054778 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:58:59.054747 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" event={"ID":"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc","Type":"ContainerStarted","Data":"06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b"} Oct 30 19:59:00.062836 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:00.062795 4789 generic.go:356] "Generic (PLEG): container finished" podID="26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" containerID="a95d19cecd90dc91e3738d0cb032d70b2b4d0fb200e9a18c5795154ad457a63b" exitCode=0 Oct 30 19:59:00.063259 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:00.062873 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" event={"ID":"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0","Type":"ContainerDied","Data":"a95d19cecd90dc91e3738d0cb032d70b2b4d0fb200e9a18c5795154ad457a63b"} Oct 30 19:59:00.064538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:00.064516 4789 generic.go:356] "Generic (PLEG): container finished" podID="bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" containerID="ace8cec909f10128d8535419172b011a71cea8c3b8c67d56401a48aabef59f19" exitCode=0 Oct 30 19:59:00.064669 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:00.064577 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" event={"ID":"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc","Type":"ContainerDied","Data":"ace8cec909f10128d8535419172b011a71cea8c3b8c67d56401a48aabef59f19"} Oct 30 19:59:01.230115 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.230088 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:59:01.250252 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.250228 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\" (UniqueName: \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\") pod \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " Oct 30 19:59:01.250349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.250260 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\") pod \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " Oct 30 19:59:01.250349 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.250326 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\") pod \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\" (UID: \"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc\") " Oct 30 19:59:01.250683 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.250643 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca" pod "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" (UID: "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:01.252342 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.252318 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca" pod "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" (UID: "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:01.252436 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.252392 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56" pod "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" (UID: "bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc"). InnerVolumeSpecName "kube-api-access-7gw56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:01.268346 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.268326 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:59:01.352023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.351957 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\" (UniqueName: \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\") pod \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " Oct 30 19:59:01.352023 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352007 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\") pod \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " Oct 30 19:59:01.352201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352031 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\") pod \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\" (UID: \"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0\") " Oct 30 19:59:01.352400 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352379 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gw56\" (UniqueName: \"kubernetes.io/projected/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-kube-api-access-7gw56\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.352502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352402 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.352502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352418 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.352502 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.352452 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca" pod "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" (UID: "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:01.354017 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.353995 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca" pod "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" (UID: "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:01.354410 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.354392 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w" pod "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" (UID: "26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0"). InnerVolumeSpecName "kube-api-access-vp72w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:01.453144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.453112 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp72w\" (UniqueName: \"kubernetes.io/projected/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-kube-api-access-vp72w\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.453144 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.453141 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.453358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:01.453155 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:02.078309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.078279 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" Oct 30 19:59:02.078309 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.078301 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8" event={"ID":"26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0","Type":"ContainerDied","Data":"f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a"} Oct 30 19:59:02.078520 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.078352 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88b74613f886acdc72d802f9ef8125e4ff7ab47eeae0f2da338a37a0a21b93a" Oct 30 19:59:02.079728 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.079707 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" Oct 30 19:59:02.079877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.079758 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q" event={"ID":"bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc","Type":"ContainerDied","Data":"06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b"} Oct 30 19:59:02.079877 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:02.079803 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f54048fd1269780ce2ffd6f802c9cc37daa8edbf78d984521bc3dfb5f5dc9b" Oct 30 19:59:04.256534 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.256470 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw"] Oct 30 19:59:04.257083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.256969 4789 cpu_manager.go:414] "containerMap: removing container" podUID="bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" containerName="revoke-keys" Oct 30 19:59:04.257083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.256982 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" containerName="revoke-keys" Oct 30 19:59:04.257083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.256992 4789 cpu_manager.go:414] "containerMap: removing container" podUID="26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" containerName="revoke-keys" Oct 30 19:59:04.257083 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.256998 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" containerName="revoke-keys" Oct 30 19:59:04.257278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.257102 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" containerName="revoke-keys" Oct 30 19:59:04.257278 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.257111 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" containerName="revoke-keys" Oct 30 19:59:04.260657 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.260637 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.314831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.314800 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j"] Oct 30 19:59:04.319388 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.319364 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.357303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.357272 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:59:04.357303 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.357294 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:59:04.359954 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.359929 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw"] Oct 30 19:59:04.361687 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.361673 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:04.361757 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.361734 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:59:04.379178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.379157 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j"] Oct 30 19:59:04.385413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385389 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.385538 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385464 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9btg\" (UniqueName: \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.385607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385549 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfq6\" (UniqueName: \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.385607 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385581 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.385716 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385617 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.385779 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.385744 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.486417 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486391 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.486562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486440 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9btg\" (UniqueName: \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.486562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486470 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfq6\" (UniqueName: \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.486562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486494 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.486562 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486519 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.486737 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.486584 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.487143 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.487117 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.487298 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.487263 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.488927 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.488901 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.489044 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.488904 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.546480 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.546422 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9btg\" (UniqueName: \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\") pod \"maas-api-revoke-sub-4e6a7660-nfhrw\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.547151 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.547133 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfq6\" (UniqueName: \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\") pod \"maas-api-revoke-sub-cd5c480a-whv9j\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.574314 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.574291 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:04.636629 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.636598 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:04.956429 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:04.956402 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw"] Oct 30 19:59:04.958554 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:59:04.958516 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6be7de_7a61_4f00_97ca_e157806ccaef.slice/crio-801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4 WatchSource:0}: Error finding container 801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4: Status 404 returned error can't find the container with id 801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4 Oct 30 19:59:05.052923 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.052898 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j"] Oct 30 19:59:05.054676 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:59:05.054643 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bf77b8_a6c1_44aa_9003_2212327d0a04.slice/crio-c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee WatchSource:0}: Error finding container c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee: Status 404 returned error can't find the container with id c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee Oct 30 19:59:05.099830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.099806 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr"] Oct 30 19:59:05.102176 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.102155 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-85f471-mrmbr"] Oct 30 19:59:05.102280 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.102194 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" event={"ID":"8d6be7de-7a61-4f00-97ca-e157806ccaef","Type":"ContainerStarted","Data":"801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4"} Oct 30 19:59:05.103526 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.103485 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" event={"ID":"c0bf77b8-a6c1-44aa-9003-2212327d0a04","Type":"ContainerStarted","Data":"c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee"} Oct 30 19:59:05.246890 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.246832 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:05.678353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:05.678258 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecd110b-dd2c-41af-bbd5-52f099ec3743" path="/var/lib/kubelet/pods/0ecd110b-dd2c-41af-bbd5-52f099ec3743/volumes" Oct 30 19:59:06.123295 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:06.123258 4789 generic.go:356] "Generic (PLEG): container finished" podID="8d6be7de-7a61-4f00-97ca-e157806ccaef" containerID="aabc23fc699bbe8df87e02670369afa122466156dbe3b81ed1a6ba70e29f757d" exitCode=0 Oct 30 19:59:06.123530 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:06.123337 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" event={"ID":"8d6be7de-7a61-4f00-97ca-e157806ccaef","Type":"ContainerDied","Data":"aabc23fc699bbe8df87e02670369afa122466156dbe3b81ed1a6ba70e29f757d"} Oct 30 19:59:06.125139 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:06.125114 4789 generic.go:356] "Generic (PLEG): container finished" podID="c0bf77b8-a6c1-44aa-9003-2212327d0a04" containerID="77f5bfa3f3f9870889403ad93294212cfb9b41d1323f943d25fe1744a3abd2c4" exitCode=0 Oct 30 19:59:06.125297 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:06.125139 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" event={"ID":"c0bf77b8-a6c1-44aa-9003-2212327d0a04","Type":"ContainerDied","Data":"77f5bfa3f3f9870889403ad93294212cfb9b41d1323f943d25fe1744a3abd2c4"} Oct 30 19:59:07.285655 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.285623 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:07.350364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.350343 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:07.429431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429359 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\") pod \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " Oct 30 19:59:07.429431 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429396 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\" (UniqueName: \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\") pod \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " Oct 30 19:59:07.429654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429533 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\") pod \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\" (UID: \"c0bf77b8-a6c1-44aa-9003-2212327d0a04\") " Oct 30 19:59:07.429654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429555 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\" (UniqueName: \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\") pod \"8d6be7de-7a61-4f00-97ca-e157806ccaef\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " Oct 30 19:59:07.429654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429608 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\") pod \"8d6be7de-7a61-4f00-97ca-e157806ccaef\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " Oct 30 19:59:07.429654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429632 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\") pod \"8d6be7de-7a61-4f00-97ca-e157806ccaef\" (UID: \"8d6be7de-7a61-4f00-97ca-e157806ccaef\") " Oct 30 19:59:07.429879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429752 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca" pod "c0bf77b8-a6c1-44aa-9003-2212327d0a04" (UID: "c0bf77b8-a6c1-44aa-9003-2212327d0a04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:07.429981 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.429959 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:07.430358 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.430334 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca" pod "8d6be7de-7a61-4f00-97ca-e157806ccaef" (UID: "8d6be7de-7a61-4f00-97ca-e157806ccaef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:07.431826 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.431803 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca" pod "c0bf77b8-a6c1-44aa-9003-2212327d0a04" (UID: "c0bf77b8-a6c1-44aa-9003-2212327d0a04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:07.431898 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.431839 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg" pod "8d6be7de-7a61-4f00-97ca-e157806ccaef" (UID: "8d6be7de-7a61-4f00-97ca-e157806ccaef"). InnerVolumeSpecName "kube-api-access-n9btg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:07.432148 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.432124 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6" pod "c0bf77b8-a6c1-44aa-9003-2212327d0a04" (UID: "c0bf77b8-a6c1-44aa-9003-2212327d0a04"). InnerVolumeSpecName "kube-api-access-6xfq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:07.432234 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.432193 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca" pod "8d6be7de-7a61-4f00-97ca-e157806ccaef" (UID: "8d6be7de-7a61-4f00-97ca-e157806ccaef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:07.530592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.530561 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bf77b8-a6c1-44aa-9003-2212327d0a04-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:07.530592 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.530593 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9btg\" (UniqueName: \"kubernetes.io/projected/8d6be7de-7a61-4f00-97ca-e157806ccaef-kube-api-access-n9btg\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:07.530783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.530603 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:07.530783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.530611 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d6be7de-7a61-4f00-97ca-e157806ccaef-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:07.530783 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:07.530620 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfq6\" (UniqueName: \"kubernetes.io/projected/c0bf77b8-a6c1-44aa-9003-2212327d0a04-kube-api-access-6xfq6\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:08.139014 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.138985 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" Oct 30 19:59:08.139201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.138985 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw" event={"ID":"8d6be7de-7a61-4f00-97ca-e157806ccaef","Type":"ContainerDied","Data":"801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4"} Oct 30 19:59:08.139201 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.139108 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="801ae60080e91081960fb318f36ef1f8db7083bf999ce00e7b737757019ad9d4" Oct 30 19:59:08.140581 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.140544 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" Oct 30 19:59:08.140709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.140590 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j" event={"ID":"c0bf77b8-a6c1-44aa-9003-2212327d0a04","Type":"ContainerDied","Data":"c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee"} Oct 30 19:59:08.140709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:08.140633 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c35879e66b8eb4ee8d7ad26063e96815ee31d2c01d5ac10301a82e333f0400ee" Oct 30 19:59:14.162957 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.162922 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9"] Oct 30 19:59:14.163376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163361 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c0bf77b8-a6c1-44aa-9003-2212327d0a04" containerName="revoke-keys" Oct 30 19:59:14.163376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163373 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bf77b8-a6c1-44aa-9003-2212327d0a04" containerName="revoke-keys" Oct 30 19:59:14.163456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163406 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8d6be7de-7a61-4f00-97ca-e157806ccaef" containerName="revoke-keys" Oct 30 19:59:14.163456 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163412 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d6be7de-7a61-4f00-97ca-e157806ccaef" containerName="revoke-keys" Oct 30 19:59:14.163553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163485 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d6be7de-7a61-4f00-97ca-e157806ccaef" containerName="revoke-keys" Oct 30 19:59:14.163553 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.163497 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bf77b8-a6c1-44aa-9003-2212327d0a04" containerName="revoke-keys" Oct 30 19:59:14.166866 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.166851 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.220238 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.220192 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:59:14.220402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.220292 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:59:14.220402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.220347 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:59:14.266548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.266521 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:14.269395 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.269370 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9"] Oct 30 19:59:14.298088 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.298058 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.300117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.298109 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.300117 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.298245 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4qc\" (UniqueName: \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.378510 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.378481 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": read tcp 10.132.0.2:54660->10.132.1.56:15021: read: connection reset by peer" start-of-body= Oct 30 19:59:14.378662 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.378550 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": read tcp 10.132.0.2:54660->10.132.1.56:15021: read: connection reset by peer" Oct 30 19:59:14.399677 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.399650 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.399770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.399693 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.399770 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.399751 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4qc\" (UniqueName: \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.400572 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.400547 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.401805 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.401777 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.457547 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.457514 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4qc\" (UniqueName: \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.481393 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.481372 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:14.858528 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:14.858503 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9"] Oct 30 19:59:14.860580 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:59:14.860542 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e30a66_8606_4e58_a915_7c3e6dcef48d.slice/crio-01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897 WatchSource:0}: Error finding container 01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897: Status 404 returned error can't find the container with id 01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897 Oct 30 19:59:15.189635 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.189603 4789 generic.go:356] "Generic (PLEG): container finished" podID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerID="9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd" exitCode=0 Oct 30 19:59:15.190011 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.189679 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerDied","Data":"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd"} Oct 30 19:59:15.190105 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.190083 4789 scope.go:122] "RemoveContainer" containerID="9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd" Oct 30 19:59:15.191157 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.191130 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" event={"ID":"49e30a66-8606-4e58-a915-7c3e6dcef48d","Type":"ContainerStarted","Data":"01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897"} Oct 30 19:59:15.191413 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.191386 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:15.191539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.191452 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:15.191539 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.191490 4789 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:15.251855 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:15.251778 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:16.213167 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.212142 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerStarted","Data":"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea"} Oct 30 19:59:16.215984 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.215947 4789 generic.go:356] "Generic (PLEG): container finished" podID="49e30a66-8606-4e58-a915-7c3e6dcef48d" containerID="ce6d8310de603d18e5e2a93a34ad829f642cf8836157b01be98e42b2b05b2a17" exitCode=0 Oct 30 19:59:16.216384 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.216354 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" event={"ID":"49e30a66-8606-4e58-a915-7c3e6dcef48d","Type":"ContainerDied","Data":"ce6d8310de603d18e5e2a93a34ad829f642cf8836157b01be98e42b2b05b2a17"} Oct 30 19:59:16.644165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.640377 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:59:16.644165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.640467 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 19:59:16.644165 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.640499 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 19:59:16.797402 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.797372 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:59:16.931978 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.931952 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\" (UniqueName: \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\") pod \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " Oct 30 19:59:16.932178 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.932167 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\") pod \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\" (UID: \"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f\") " Oct 30 19:59:16.932636 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.932610 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume" pod "ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" (UID: "ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:16.934244 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:16.934220 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz" pod "ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" (UID: "ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f"). InnerVolumeSpecName "kube-api-access-hm5qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:17.033606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.033521 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm5qz\" (UniqueName: \"kubernetes.io/projected/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-kube-api-access-hm5qz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:17.033606 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.033546 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:17.226142 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.226108 4789 generic.go:356] "Generic (PLEG): container finished" podID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerID="7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3" exitCode=0 Oct 30 19:59:17.226602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.226168 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" Oct 30 19:59:17.226602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.226196 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" event={"ID":"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f","Type":"ContainerDied","Data":"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3"} Oct 30 19:59:17.226602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.226270 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2" event={"ID":"ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f","Type":"ContainerDied","Data":"09d744add1ac816bf638580c3ae0a51ce9f44eabd3c5cb7c15139b428c7d47ab"} Oct 30 19:59:17.226602 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.226290 4789 scope.go:122] "RemoveContainer" containerID="7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3" Oct 30 19:59:17.238110 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.238038 4789 scope.go:122] "RemoveContainer" containerID="7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3" Oct 30 19:59:17.238367 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:59:17.238339 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3\": container with ID starting with 7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3 not found: ID does not exist" containerID="7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3" Oct 30 19:59:17.238457 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.238380 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3"} err="failed to get container status \"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3\": rpc error: code = NotFound desc = could not find container \"7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3\": container with ID starting with 7263580b3810bc6fb308b49a2bcd9f45038eaab32890dc8369410b7aa3d884a3 not found: ID does not exist" Oct 30 19:59:17.378864 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.378832 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:17.384830 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.384808 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:59:17.418390 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.418322 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-2301cb-67d757b85f-ks9b2"] Oct 30 19:59:17.436905 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.436884 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\") pod \"49e30a66-8606-4e58-a915-7c3e6dcef48d\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " Oct 30 19:59:17.437033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.436933 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\") pod \"49e30a66-8606-4e58-a915-7c3e6dcef48d\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " Oct 30 19:59:17.437033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.437025 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\" (UniqueName: \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\") pod \"49e30a66-8606-4e58-a915-7c3e6dcef48d\" (UID: \"49e30a66-8606-4e58-a915-7c3e6dcef48d\") " Oct 30 19:59:17.437353 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.437328 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca" pod "49e30a66-8606-4e58-a915-7c3e6dcef48d" (UID: "49e30a66-8606-4e58-a915-7c3e6dcef48d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:17.437461 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.437448 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:17.438995 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.438967 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc" pod "49e30a66-8606-4e58-a915-7c3e6dcef48d" (UID: "49e30a66-8606-4e58-a915-7c3e6dcef48d"). InnerVolumeSpecName "kube-api-access-bk4qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:17.439089 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.438997 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca" pod "49e30a66-8606-4e58-a915-7c3e6dcef48d" (UID: "49e30a66-8606-4e58-a915-7c3e6dcef48d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:17.539245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.538140 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e30a66-8606-4e58-a915-7c3e6dcef48d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:17.539245 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.538173 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk4qc\" (UniqueName: \"kubernetes.io/projected/49e30a66-8606-4e58-a915-7c3e6dcef48d-kube-api-access-bk4qc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:17.640005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.639931 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 19:59:17.640005 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.639989 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 19:59:17.668518 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:17.668489 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" path="/var/lib/kubelet/pods/ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f/volumes" Oct 30 19:59:18.222065 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:59:18.222029 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:59:18.235225 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:18.235188 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" Oct 30 19:59:18.235651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:18.235247 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9" event={"ID":"49e30a66-8606-4e58-a915-7c3e6dcef48d","Type":"ContainerDied","Data":"01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897"} Oct 30 19:59:18.235651 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:18.235318 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e578754a058fae678251646c2734843b9e9e79a19ab6d212ce8ae726c6f897" Oct 30 19:59:18.639879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:18.639808 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 19:59:18.639879 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:18.639863 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 19:59:19.640458 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:19.640418 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 19:59:19.640860 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:19.640476 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 19:59:20.639746 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:20.639712 4789 patch_prober.go:29] interesting pod/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body= Oct 30 19:59:20.639959 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:20.639771 4789 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" Oct 30 19:59:21.644364 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:21.644333 4789 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:59:21.644832 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:21.644811 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:59:21.645591 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:21.645569 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 19:59:26.556764 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:26.556728 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr"] Oct 30 19:59:26.602548 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:26.602521 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-22fb18-4hlxr"] Oct 30 19:59:27.667141 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:27.667105 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73b5b5e-e056-4973-8aa4-31656aa11983" path="/var/lib/kubelet/pods/d73b5b5e-e056-4973-8aa4-31656aa11983/volumes" Oct 30 19:59:28.760182 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:28.760149 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:59:28.858445 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:28.858417 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:59:33.761027 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:33.760976 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" containerID="cri-o://d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474" gracePeriod=25 Oct 30 19:59:33.859465 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:33.859435 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" containerID="cri-o://3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707" gracePeriod=25 Oct 30 19:59:37.513858 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:37.513818 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx"] Oct 30 19:59:37.559773 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:37.559740 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-2301cb-bhdcx"] Oct 30 19:59:37.674709 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:37.668387 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962ef5a2-88fb-4943-9b97-09c8c8865bf0" path="/var/lib/kubelet/pods/962ef5a2-88fb-4943-9b97-09c8c8865bf0/volumes" Oct 30 19:59:38.658193 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:38.658156 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5"] Oct 30 19:59:38.704305 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:38.704277 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9275c3-nxpt5"] Oct 30 19:59:39.666949 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:39.666917 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6922fa7-e81d-4ea1-9384-12308de983de" path="/var/lib/kubelet/pods/a6922fa7-e81d-4ea1-9384-12308de983de/volumes" Oct 30 19:59:42.535015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:42.534972 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:42.535015 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:42.534994 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:48.197710 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:59:48.197669 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 19:59:49.513173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:49.513139 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9"] Oct 30 19:59:49.558670 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:49.558646 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c01685-w5km9"] Oct 30 19:59:49.667704 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:49.667677 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e30a66-8606-4e58-a915-7c3e6dcef48d" path="/var/lib/kubelet/pods/49e30a66-8606-4e58-a915-7c3e6dcef48d/volumes" Oct 30 19:59:52.535507 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:52.535460 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:52.536007 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:52.535461 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:54.013614 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.013581 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:59:54.085892 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.085865 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\") pod \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " Oct 30 19:59:54.086041 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.085969 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\" (UniqueName: \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\") pod \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\" (UID: \"01327fa7-0b30-4de6-b800-3efb3b4d9d45\") " Oct 30 19:59:54.086312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.086284 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume" pod "01327fa7-0b30-4de6-b800-3efb3b4d9d45" (UID: "01327fa7-0b30-4de6-b800-3efb3b4d9d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:54.088099 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.088072 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn" pod "01327fa7-0b30-4de6-b800-3efb3b4d9d45" (UID: "01327fa7-0b30-4de6-b800-3efb3b4d9d45"). InnerVolumeSpecName "kube-api-access-n2wcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:54.105878 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.105856 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:59:54.187204 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.187178 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\" (UniqueName: \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\") pod \"f299e81e-59da-447e-8d2d-b6260c50831e\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " Oct 30 19:59:54.187376 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.187355 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\") pod \"f299e81e-59da-447e-8d2d-b6260c50831e\" (UID: \"f299e81e-59da-447e-8d2d-b6260c50831e\") " Oct 30 19:59:54.187641 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.187623 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01327fa7-0b30-4de6-b800-3efb3b4d9d45-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:54.187714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.187644 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wcn\" (UniqueName: \"kubernetes.io/projected/01327fa7-0b30-4de6-b800-3efb3b4d9d45-kube-api-access-n2wcn\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:54.187714 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.187697 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume" pod "f299e81e-59da-447e-8d2d-b6260c50831e" (UID: "f299e81e-59da-447e-8d2d-b6260c50831e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:54.189173 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.189145 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw" pod "f299e81e-59da-447e-8d2d-b6260c50831e" (UID: "f299e81e-59da-447e-8d2d-b6260c50831e"). InnerVolumeSpecName "kube-api-access-8gtnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:54.288649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.288593 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gtnw\" (UniqueName: \"kubernetes.io/projected/f299e81e-59da-447e-8d2d-b6260c50831e-kube-api-access-8gtnw\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:54.288649 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.288616 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f299e81e-59da-447e-8d2d-b6260c50831e-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:54.481293 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.481263 4789 generic.go:356] "Generic (PLEG): container finished" podID="f299e81e-59da-447e-8d2d-b6260c50831e" containerID="3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707" exitCode=0 Oct 30 19:59:54.481443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.481316 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" Oct 30 19:59:54.481443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.481338 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" event={"ID":"f299e81e-59da-447e-8d2d-b6260c50831e","Type":"ContainerDied","Data":"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707"} Oct 30 19:59:54.481443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.481390 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5" event={"ID":"f299e81e-59da-447e-8d2d-b6260c50831e","Type":"ContainerDied","Data":"c68932d404150c64d91125cd08d02c039a669c1cb8114fbddd1618c089fb4be3"} Oct 30 19:59:54.481443 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.481415 4789 scope.go:122] "RemoveContainer" containerID="3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707" Oct 30 19:59:54.482965 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.482926 4789 generic.go:356] "Generic (PLEG): container finished" podID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerID="d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474" exitCode=0 Oct 30 19:59:54.483077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.482955 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" event={"ID":"01327fa7-0b30-4de6-b800-3efb3b4d9d45","Type":"ContainerDied","Data":"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474"} Oct 30 19:59:54.483077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.483011 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" event={"ID":"01327fa7-0b30-4de6-b800-3efb3b4d9d45","Type":"ContainerDied","Data":"fc8ce93ea3e0a7bc4e9d24d8bcde702f13c6bc64ac8b196d304bd64c0d728f48"} Oct 30 19:59:54.483077 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.482970 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm" Oct 30 19:59:54.492478 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.492460 4789 scope.go:122] "RemoveContainer" containerID="3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707" Oct 30 19:59:54.492831 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:59:54.492741 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707\": container with ID starting with 3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707 not found: ID does not exist" containerID="3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707" Oct 30 19:59:54.492831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.492766 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707"} err="failed to get container status \"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707\": rpc error: code = NotFound desc = could not find container \"3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707\": container with ID starting with 3faea3df6e9d0d67c77975ee39d7b31f50b5664e74e2e6c5cbafa92eb9e65707 not found: ID does not exist" Oct 30 19:59:54.492831 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.492782 4789 scope.go:122] "RemoveContainer" containerID="d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474" Oct 30 19:59:54.501371 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.501353 4789 scope.go:122] "RemoveContainer" containerID="d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474" Oct 30 19:59:54.501610 ip-10-0-136-46 kubenswrapper[4789]: E1030 19:59:54.501593 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474\": container with ID starting with d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474 not found: ID does not exist" containerID="d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474" Oct 30 19:59:54.501654 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.501617 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474"} err="failed to get container status \"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474\": rpc error: code = NotFound desc = could not find container \"d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474\": container with ID starting with d26906cfe5a5d763c9b4c39f2496353fa65f8f913a3a1e0eb871462af672a474 not found: ID does not exist" Oct 30 19:59:54.638873 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.638821 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:59:54.641033 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.641016 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c01685-55cbfd57cf-52jn5"] Oct 30 19:59:54.744391 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.744358 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:59:54.746765 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:54.746746 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c01685-b875f45bd-djpvm"] Oct 30 19:59:55.667381 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:55.667347 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" path="/var/lib/kubelet/pods/01327fa7-0b30-4de6-b800-3efb3b4d9d45/volumes" Oct 30 19:59:55.667894 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:55.667873 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" path="/var/lib/kubelet/pods/f299e81e-59da-447e-8d2d-b6260c50831e/volumes" Oct 30 19:59:58.175963 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.175929 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4"] Oct 30 19:59:58.176387 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176369 4789 cpu_manager.go:414] "containerMap: removing container" podUID="49e30a66-8606-4e58-a915-7c3e6dcef48d" containerName="revoke-keys" Oct 30 19:59:58.176434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176389 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e30a66-8606-4e58-a915-7c3e6dcef48d" containerName="revoke-keys" Oct 30 19:59:58.176434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176408 4789 cpu_manager.go:414] "containerMap: removing container" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" Oct 30 19:59:58.176434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176414 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" Oct 30 19:59:58.176434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176425 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" Oct 30 19:59:58.176434 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176431 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176446 4789 cpu_manager.go:414] "containerMap: removing container" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176453 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176523 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="01327fa7-0b30-4de6-b800-3efb3b4d9d45" containerName="payload-processing" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176539 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e30a66-8606-4e58-a915-7c3e6dcef48d" containerName="revoke-keys" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176548 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf0c815-d37d-4f09-8e4b-1fbd65d5dd7f" containerName="payload-pre-processing" Oct 30 19:59:58.176604 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.176556 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f299e81e-59da-447e-8d2d-b6260c50831e" containerName="payload-pre-processing" Oct 30 19:59:58.181261 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.181241 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.222419 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.222394 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qjs\" (UniqueName: \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.222541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.222426 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.222541 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.222450 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.235352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.235327 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:59:58.235352 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.235338 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:59:58.239430 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.239412 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:58.281749 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.281721 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 19:59:58.284008 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.283989 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4"] Oct 30 19:59:58.323174 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.323147 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qjs\" (UniqueName: \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.323312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.323191 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.323312 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.323257 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.323844 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.323820 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.325519 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.325498 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.382424 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.382399 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qjs\" (UniqueName: \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\") pod \"maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.493994 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.493970 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 19:59:58.867819 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:58.867792 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4"] Oct 30 19:59:58.869491 ip-10-0-136-46 kubenswrapper[4789]: W1030 19:59:58.869463 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c74e44_3a6e_4b0b_956c_e125cf1979a8.slice/crio-b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7 WatchSource:0}: Error finding container b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7: Status 404 returned error can't find the container with id b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7 Oct 30 19:59:59.525763 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:59.525683 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" event={"ID":"f5c74e44-3a6e-4b0b-956c-e125cf1979a8","Type":"ContainerStarted","Data":"3e0a6f144c2133868ed7e57f6a583be7edf25b1f37d7348c830f72a11512f0ce"} Oct 30 19:59:59.526113 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:59.525784 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" event={"ID":"f5c74e44-3a6e-4b0b-956c-e125cf1979a8","Type":"ContainerStarted","Data":"b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7"} Oct 30 19:59:59.636988 ip-10-0-136-46 kubenswrapper[4789]: I1030 19:59:59.636935 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" podStartSLOduration=1.636923204 podStartE2EDuration="1.636923204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:59.636020143 +0000 UTC m=+4436.137930632" watchObservedRunningTime="2026-10-30 19:59:59.636923204 +0000 UTC m=+4436.138833689" Oct 30 20:00:00.171403 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.171371 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:00.175187 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.175167 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.183437 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.183414 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:00.187398 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.187383 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.243600 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.243568 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.243705 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.243610 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.243750 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.243697 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxlz\" (UniqueName: \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.243792 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.243774 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tvf\" (UniqueName: \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.270984 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.270959 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:00.275984 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.275960 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:00.344759 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.344718 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxlz\" (UniqueName: \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.344880 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.344806 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tvf\" (UniqueName: \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.344880 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.344861 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.344967 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.344890 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.345536 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.345516 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.345702 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.345677 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.403246 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.403190 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxlz\" (UniqueName: \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\") pod \"maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.403246 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.403238 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tvf\" (UniqueName: \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\") pod \"maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.489048 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.489027 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:00.499253 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.499236 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:00.536820 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.536786 4789 generic.go:356] "Generic (PLEG): container finished" podID="f5c74e44-3a6e-4b0b-956c-e125cf1979a8" containerID="3e0a6f144c2133868ed7e57f6a583be7edf25b1f37d7348c830f72a11512f0ce" exitCode=0 Oct 30 20:00:00.537250 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.536946 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" event={"ID":"f5c74e44-3a6e-4b0b-956c-e125cf1979a8","Type":"ContainerDied","Data":"3e0a6f144c2133868ed7e57f6a583be7edf25b1f37d7348c830f72a11512f0ce"} Oct 30 20:00:00.862672 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.862648 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:00.864673 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:00:00.864642 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49b3d1e_1ee1_462a_80b2_7205b8063197.slice/crio-e9fdb3a7ca7376c98b67a0684f5539ffe35cfc95a14e93bb6877727749fa7ff9 WatchSource:0}: Error finding container e9fdb3a7ca7376c98b67a0684f5539ffe35cfc95a14e93bb6877727749fa7ff9: Status 404 returned error can't find the container with id e9fdb3a7ca7376c98b67a0684f5539ffe35cfc95a14e93bb6877727749fa7ff9 Oct 30 20:00:00.913026 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:00.912959 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:00.915158 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:00:00.915135 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode809529b_0e7c_4a1d_b907_55043126357d.slice/crio-9479cbc04f838c1492c96c915243315b781ed0c4ee44cf3aaca1047c986cb00b WatchSource:0}: Error finding container 9479cbc04f838c1492c96c915243315b781ed0c4ee44cf3aaca1047c986cb00b: Status 404 returned error can't find the container with id 9479cbc04f838c1492c96c915243315b781ed0c4ee44cf3aaca1047c986cb00b Oct 30 20:00:01.544792 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.544691 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerStarted","Data":"f96e54c69a2f91d01f74f5d74a94d74684e0f5952011927573db488de9c29c73"} Oct 30 20:00:01.544792 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.544768 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerStarted","Data":"e9fdb3a7ca7376c98b67a0684f5539ffe35cfc95a14e93bb6877727749fa7ff9"} Oct 30 20:00:01.546556 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.546528 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerStarted","Data":"c84af9f5afdb2b21166249b92a7845218229e239555f9bcd688a3ffad534a496"} Oct 30 20:00:01.546708 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.546571 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerStarted","Data":"9479cbc04f838c1492c96c915243315b781ed0c4ee44cf3aaca1047c986cb00b"} Oct 30 20:00:01.659819 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.659700 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" podStartSLOduration=1.659682117 podStartE2EDuration="1.659682117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.657283485 +0000 UTC m=+4438.159193975" watchObservedRunningTime="2026-10-30 20:00:01.659682117 +0000 UTC m=+4438.161592602" Oct 30 20:00:01.682541 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.682516 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 20:00:01.757944 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.757921 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\") pod \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " Oct 30 20:00:01.758065 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.758024 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\" (UniqueName: \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\") pod \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " Oct 30 20:00:01.758120 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.758073 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\") pod \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\" (UID: \"f5c74e44-3a6e-4b0b-956c-e125cf1979a8\") " Oct 30 20:00:01.758485 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.758455 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca" pod "f5c74e44-3a6e-4b0b-956c-e125cf1979a8" (UID: "f5c74e44-3a6e-4b0b-956c-e125cf1979a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:01.759874 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.759849 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca" pod "f5c74e44-3a6e-4b0b-956c-e125cf1979a8" (UID: "f5c74e44-3a6e-4b0b-956c-e125cf1979a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:01.759998 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.759973 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs" pod "f5c74e44-3a6e-4b0b-956c-e125cf1979a8" (UID: "f5c74e44-3a6e-4b0b-956c-e125cf1979a8"). InnerVolumeSpecName "kube-api-access-h2qjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:01.769303 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.769256 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" podStartSLOduration=1.7692399079999999 podStartE2EDuration="1.769239908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.767188868 +0000 UTC m=+4438.269099353" watchObservedRunningTime="2026-10-30 20:00:01.769239908 +0000 UTC m=+4438.271150390" Oct 30 20:00:01.859322 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.859300 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qjs\" (UniqueName: \"kubernetes.io/projected/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-kube-api-access-h2qjs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.859322 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.859322 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.859465 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:01.859331 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c74e44-3a6e-4b0b-956c-e125cf1979a8-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:02.553882 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.553851 4789 generic.go:356] "Generic (PLEG): container finished" podID="e809529b-0e7c-4a1d-b907-55043126357d" containerID="c84af9f5afdb2b21166249b92a7845218229e239555f9bcd688a3ffad534a496" exitCode=60 Oct 30 20:00:02.554261 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.553929 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerDied","Data":"c84af9f5afdb2b21166249b92a7845218229e239555f9bcd688a3ffad534a496"} Oct 30 20:00:02.554261 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.554098 4789 scope.go:122] "RemoveContainer" containerID="c84af9f5afdb2b21166249b92a7845218229e239555f9bcd688a3ffad534a496" Oct 30 20:00:02.555338 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.555317 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" Oct 30 20:00:02.555447 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.555322 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4" event={"ID":"f5c74e44-3a6e-4b0b-956c-e125cf1979a8","Type":"ContainerDied","Data":"b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7"} Oct 30 20:00:02.555447 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.555443 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f08c0a516fd8af9a40f5afc62f12627628a570f92e6ccf1cbd181de33701a7" Oct 30 20:00:02.557226 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.557066 4789 generic.go:356] "Generic (PLEG): container finished" podID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerID="f96e54c69a2f91d01f74f5d74a94d74684e0f5952011927573db488de9c29c73" exitCode=60 Oct 30 20:00:02.557226 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.557095 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerDied","Data":"f96e54c69a2f91d01f74f5d74a94d74684e0f5952011927573db488de9c29c73"} Oct 30 20:00:02.557352 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.557300 4789 scope.go:122] "RemoveContainer" containerID="f96e54c69a2f91d01f74f5d74a94d74684e0f5952011927573db488de9c29c73" Oct 30 20:00:02.649329 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.649302 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 20:00:02.677118 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:02.677063 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 20:00:03.566521 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.566494 4789 generic.go:356] "Generic (PLEG): container finished" podID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerID="9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616" exitCode=60 Oct 30 20:00:03.566904 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.566580 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerDied","Data":"9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616"} Oct 30 20:00:03.566904 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.566686 4789 scope.go:122] "RemoveContainer" containerID="f96e54c69a2f91d01f74f5d74a94d74684e0f5952011927573db488de9c29c73" Oct 30 20:00:03.566988 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.566912 4789 scope.go:122] "RemoveContainer" containerID="9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616" Oct 30 20:00:03.567186 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:03.567165 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb_odh-ai-gateway-infra(b49b3d1e-1ee1-462a-80b2-7205b8063197)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" Oct 30 20:00:03.568703 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.568681 4789 generic.go:356] "Generic (PLEG): container finished" podID="e809529b-0e7c-4a1d-b907-55043126357d" containerID="67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c" exitCode=60 Oct 30 20:00:03.568813 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.568757 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerDied","Data":"67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c"} Oct 30 20:00:03.568971 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.568957 4789 scope.go:122] "RemoveContainer" containerID="67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c" Oct 30 20:00:03.569171 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:03.569155 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv_odh-ai-gateway-infra(e809529b-0e7c-4a1d-b907-55043126357d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" podUID="e809529b-0e7c-4a1d-b907-55043126357d" Oct 30 20:00:03.578445 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:03.578426 4789 scope.go:122] "RemoveContainer" containerID="c84af9f5afdb2b21166249b92a7845218229e239555f9bcd688a3ffad534a496" Oct 30 20:00:04.576147 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:04.576114 4789 scope.go:122] "RemoveContainer" containerID="67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c" Oct 30 20:00:04.576617 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:04.576370 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv_odh-ai-gateway-infra(e809529b-0e7c-4a1d-b907-55043126357d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" podUID="e809529b-0e7c-4a1d-b907-55043126357d" Oct 30 20:00:04.577976 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:04.577957 4789 scope.go:122] "RemoveContainer" containerID="9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616" Oct 30 20:00:04.578151 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:04.578134 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb_odh-ai-gateway-infra(b49b3d1e-1ee1-462a-80b2-7205b8063197)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" Oct 30 20:00:07.671406 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:07.671373 4789 scope.go:122] "RemoveContainer" containerID="9c9e960d03f867d18ac1225a2dde4f80055d924dfa891596f1f26166a647cbc2" Oct 30 20:00:14.661074 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:14.661005 4789 scope.go:122] "RemoveContainer" containerID="67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c" Oct 30 20:00:15.648796 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:15.648761 4789 generic.go:356] "Generic (PLEG): container finished" podID="e809529b-0e7c-4a1d-b907-55043126357d" containerID="b0c67fbed7e0bcd516d44b89f05cfaf396a7a691ecae1f08c3cd8efbf3896c68" exitCode=60 Oct 30 20:00:15.648921 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:15.648833 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerDied","Data":"b0c67fbed7e0bcd516d44b89f05cfaf396a7a691ecae1f08c3cd8efbf3896c68"} Oct 30 20:00:15.648921 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:15.648898 4789 scope.go:122] "RemoveContainer" containerID="67d23351e700432b02fba441dee7442b34124efbaadc5987041ce3bcbb69537c" Oct 30 20:00:15.649219 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:15.649185 4789 scope.go:122] "RemoveContainer" containerID="b0c67fbed7e0bcd516d44b89f05cfaf396a7a691ecae1f08c3cd8efbf3896c68" Oct 30 20:00:15.649521 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:15.649496 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv_odh-ai-gateway-infra(e809529b-0e7c-4a1d-b907-55043126357d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" podUID="e809529b-0e7c-4a1d-b907-55043126357d" Oct 30 20:00:16.660848 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:16.660821 4789 scope.go:122] "RemoveContainer" containerID="9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616" Oct 30 20:00:16.769016 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:16.768991 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:16.925466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:16.925444 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:17.002136 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.002114 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\") pod \"e809529b-0e7c-4a1d-b907-55043126357d\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " Oct 30 20:00:17.002284 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.002268 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\" (UniqueName: \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\") pod \"e809529b-0e7c-4a1d-b907-55043126357d\" (UID: \"e809529b-0e7c-4a1d-b907-55043126357d\") " Oct 30 20:00:17.002451 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.002429 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca" pod "e809529b-0e7c-4a1d-b907-55043126357d" (UID: "e809529b-0e7c-4a1d-b907-55043126357d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.002593 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.002580 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e809529b-0e7c-4a1d-b907-55043126357d-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.004193 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.004164 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf" pod "e809529b-0e7c-4a1d-b907-55043126357d" (UID: "e809529b-0e7c-4a1d-b907-55043126357d"). InnerVolumeSpecName "kube-api-access-l8tvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.103793 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.103765 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tvf\" (UniqueName: \"kubernetes.io/projected/e809529b-0e7c-4a1d-b907-55043126357d-kube-api-access-l8tvf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.606683 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.606653 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw"] Oct 30 20:00:17.607071 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607058 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607122 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607072 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607122 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607089 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607122 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607094 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607122 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607104 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f5c74e44-3a6e-4b0b-956c-e125cf1979a8" containerName="revoke-keys" Oct 30 20:00:17.607122 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607110 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c74e44-3a6e-4b0b-956c-e125cf1979a8" containerName="revoke-keys" Oct 30 20:00:17.607315 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607183 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607315 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607193 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.607315 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.607201 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c74e44-3a6e-4b0b-956c-e125cf1979a8" containerName="revoke-keys" Oct 30 20:00:17.610267 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.610250 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.665629 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.665602 4789 generic.go:356] "Generic (PLEG): container finished" podID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerID="8195aa373f5ec32b26fadf9fb972a9123b122b94c34e2fa371ab92f0234e34d8" exitCode=60 Oct 30 20:00:17.668001 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.667978 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:17.669089 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.669067 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj"] Oct 30 20:00:17.669548 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.669531 4789 cpu_manager.go:414] "containerMap: removing container" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.669548 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.669549 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.669678 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.669647 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="e809529b-0e7c-4a1d-b907-55043126357d" containerName="cleanup" Oct 30 20:00:17.670182 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.670133 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerName="main" containerID="cri-o://690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5" gracePeriod=45 Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.673062 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerDied","Data":"8195aa373f5ec32b26fadf9fb972a9123b122b94c34e2fa371ab92f0234e34d8"} Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.673104 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" event={"ID":"e809529b-0e7c-4a1d-b907-55043126357d","Type":"ContainerDied","Data":"9479cbc04f838c1492c96c915243315b781ed0c4ee44cf3aaca1047c986cb00b"} Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.673137 4789 scope.go:122] "RemoveContainer" containerID="9cb2cd40cea71c73bf7b24fa53b95d23138224a950d1d17a14f108f82a54e616" Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.673491 4789 scope.go:122] "RemoveContainer" containerID="8195aa373f5ec32b26fadf9fb972a9123b122b94c34e2fa371ab92f0234e34d8" Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:17.673716 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb_odh-ai-gateway-infra(b49b3d1e-1ee1-462a-80b2-7205b8063197)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" Oct 30 20:00:17.674330 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.674075 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.698613 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.698569 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" containerID="cri-o://d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0" gracePeriod=45 Oct 30 20:00:17.707650 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.707632 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:17.707920 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.707893 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pdf\" (UniqueName: \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.707971 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.707953 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.708135 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.708111 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.710321 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.710298 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw"] Oct 30 20:00:17.727139 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.727120 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj"] Oct 30 20:00:17.809080 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809053 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsvs\" (UniqueName: \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.809225 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809094 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.809381 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809252 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.809381 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809289 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pdf\" (UniqueName: \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.809381 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809327 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.809546 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809383 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.809851 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.809830 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.817629 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.817605 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.822030 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.821719 4789 scope.go:122] "RemoveContainer" containerID="b0c67fbed7e0bcd516d44b89f05cfaf396a7a691ecae1f08c3cd8efbf3896c68" Oct 30 20:00:17.872833 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.872724 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pdf\" (UniqueName: \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\") pod \"maas-api-revoke-sub-bdb91441-6f6mw\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.910141 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.910111 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsvs\" (UniqueName: \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.910296 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.910202 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.910360 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.910306 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.910915 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.910895 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.912432 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.912412 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.923430 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.923409 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:17.944928 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.944900 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 20:00:17.972062 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.972027 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsvs\" (UniqueName: \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\") pod \"maas-api-revoke-sub-bf06d798-kqzsj\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:17.988453 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:17.988426 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 20:00:18.112949 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.112873 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.112949 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.112917 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.113167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.112975 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\" (UniqueName: \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.113167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113006 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.113167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113046 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113087 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113159 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113233 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113281 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113302 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\" (UniqueName: \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113334 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\") pod \"8d69644f-e496-43f7-aa37-6264364d8078\" (UID: \"8d69644f-e496-43f7-aa37-6264364d8078\") " Oct 30 20:00:18.113419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113378 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113443 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113448 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113481 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\") pod \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\" (UID: \"c2db97b9-2af8-4f06-84d4-3cff4a5854ac\") " Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113530 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113666 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.113754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113697 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113885 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113909 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113925 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113939 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113952 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.114093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.113966 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.116354 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.115929 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "kube-api-access-fn8xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.116909 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.116382 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.117133 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.117083 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:18.117265 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.117130 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs" pod "8d69644f-e496-43f7-aa37-6264364d8078" (UID: "8d69644f-e496-43f7-aa37-6264364d8078"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:18.117463 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.117421 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "kube-api-access-2t9xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.120162 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.120136 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm" pod "c2db97b9-2af8-4f06-84d4-3cff4a5854ac" (UID: "c2db97b9-2af8-4f06-84d4-3cff4a5854ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.137334 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.137314 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:18.205514 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:18.201564 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:00:18.214964 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214939 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.214964 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214966 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.215096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214975 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8xv\" (UniqueName: \"kubernetes.io/projected/8d69644f-e496-43f7-aa37-6264364d8078-kube-api-access-fn8xv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.215096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214983 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d69644f-e496-43f7-aa37-6264364d8078-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.215096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214992 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9xb\" (UniqueName: \"kubernetes.io/projected/c2db97b9-2af8-4f06-84d4-3cff4a5854ac-kube-api-access-2t9xb\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.215096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.214999 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d69644f-e496-43f7-aa37-6264364d8078-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.311593 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.311566 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw"] Oct 30 20:00:18.314053 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:00:18.314008 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6787b8_a665_45b7_8b2f_52894a53f21d.slice/crio-a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64 WatchSource:0}: Error finding container a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64: Status 404 returned error can't find the container with id a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64 Oct 30 20:00:18.513259 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.513231 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj"] Oct 30 20:00:18.516449 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:00:18.516416 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78148758_1e4e_4117_b8d2_5ad6e83e28d5.slice/crio-ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a WatchSource:0}: Error finding container ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a: Status 404 returned error can't find the container with id ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a Oct 30 20:00:18.677514 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.677486 4789 generic.go:356] "Generic (PLEG): container finished" podID="8d69644f-e496-43f7-aa37-6264364d8078" containerID="d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0" exitCode=0 Oct 30 20:00:18.677910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.677522 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" Oct 30 20:00:18.677910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.677567 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" event={"ID":"8d69644f-e496-43f7-aa37-6264364d8078","Type":"ContainerDied","Data":"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0"} Oct 30 20:00:18.677910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.677607 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk" event={"ID":"8d69644f-e496-43f7-aa37-6264364d8078","Type":"ContainerDied","Data":"d2a04fd931ad850660e737be035184e7379510417794576913cbc6b0cfc9a018"} Oct 30 20:00:18.677910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.677628 4789 scope.go:122] "RemoveContainer" containerID="d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0" Oct 30 20:00:18.680805 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.680766 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" event={"ID":"78148758-1e4e-4117-b8d2-5ad6e83e28d5","Type":"ContainerStarted","Data":"ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a"} Oct 30 20:00:18.682754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.682698 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" event={"ID":"4a6787b8-a665-45b7-8b2f-52894a53f21d","Type":"ContainerStarted","Data":"844a6be0979dfa0ec64a31a5b420878cd5429cffcf211c24f63591bcee04fa18"} Oct 30 20:00:18.682754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.682743 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" event={"ID":"4a6787b8-a665-45b7-8b2f-52894a53f21d","Type":"ContainerStarted","Data":"a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64"} Oct 30 20:00:18.684493 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.684467 4789 generic.go:356] "Generic (PLEG): container finished" podID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerID="690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5" exitCode=0 Oct 30 20:00:18.684599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.684506 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" Oct 30 20:00:18.684599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.684519 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" event={"ID":"c2db97b9-2af8-4f06-84d4-3cff4a5854ac","Type":"ContainerDied","Data":"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5"} Oct 30 20:00:18.684599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.684560 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5" event={"ID":"c2db97b9-2af8-4f06-84d4-3cff4a5854ac","Type":"ContainerDied","Data":"0e6fecb6f5326303c7181d9b61eeb671a69acc485f2f03feb1ad2ab47c57be03"} Oct 30 20:00:18.712748 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.712713 4789 scope.go:122] "RemoveContainer" containerID="d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0" Oct 30 20:00:18.713024 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:18.713005 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0\": container with ID starting with d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0 not found: ID does not exist" containerID="d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0" Oct 30 20:00:18.713099 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.713033 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0"} err="failed to get container status \"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0\": rpc error: code = NotFound desc = could not find container \"d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0\": container with ID starting with d123048e04b2b53a06762bf1b91f3a0471bf1d562ba4fc1577f369799204b6b0 not found: ID does not exist" Oct 30 20:00:18.713099 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.713050 4789 scope.go:122] "RemoveContainer" containerID="690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5" Oct 30 20:00:18.724536 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.724515 4789 scope.go:122] "RemoveContainer" containerID="690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5" Oct 30 20:00:18.724832 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:18.724809 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5\": container with ID starting with 690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5 not found: ID does not exist" containerID="690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5" Oct 30 20:00:18.724913 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.724842 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5"} err="failed to get container status \"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5\": rpc error: code = NotFound desc = could not find container \"690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5\": container with ID starting with 690a07a6b187cdb055e47004dd07af71b553d5a8f8f81d0a57588f4448125af5 not found: ID does not exist" Oct 30 20:00:18.842479 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.842393 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" podStartSLOduration=1.842380258 podStartE2EDuration="1.842380258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:18.839997435 +0000 UTC m=+4455.341907931" watchObservedRunningTime="2026-10-30 20:00:18.842380258 +0000 UTC m=+4455.344290740" Oct 30 20:00:18.847575 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.847550 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:18.951832 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.951803 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 20:00:18.957453 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.957421 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-330025/simulator-w2-330025-kserve-c5c76cf84-pmskk"] Oct 30 20:00:18.992919 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:18.992898 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:19.061071 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.061047 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 20:00:19.066263 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.066238 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-330025/distinct-2-w2-330025-kserve-78987cff7f-czss5"] Oct 30 20:00:19.124044 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.123994 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\") pod \"b49b3d1e-1ee1-462a-80b2-7205b8063197\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " Oct 30 20:00:19.124140 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.124065 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\" (UniqueName: \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\") pod \"b49b3d1e-1ee1-462a-80b2-7205b8063197\" (UID: \"b49b3d1e-1ee1-462a-80b2-7205b8063197\") " Oct 30 20:00:19.124363 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.124343 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca" pod "b49b3d1e-1ee1-462a-80b2-7205b8063197" (UID: "b49b3d1e-1ee1-462a-80b2-7205b8063197"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.126080 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.126057 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz" pod "b49b3d1e-1ee1-462a-80b2-7205b8063197" (UID: "b49b3d1e-1ee1-462a-80b2-7205b8063197"). InnerVolumeSpecName "kube-api-access-6zxlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.224974 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.224950 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49b3d1e-1ee1-462a-80b2-7205b8063197-maas-api-service-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.224974 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.224973 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxlz\" (UniqueName: \"kubernetes.io/projected/b49b3d1e-1ee1-462a-80b2-7205b8063197-kube-api-access-6zxlz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.666983 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.666954 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d69644f-e496-43f7-aa37-6264364d8078" path="/var/lib/kubelet/pods/8d69644f-e496-43f7-aa37-6264364d8078/volumes" Oct 30 20:00:19.667359 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.667345 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" path="/var/lib/kubelet/pods/c2db97b9-2af8-4f06-84d4-3cff4a5854ac/volumes" Oct 30 20:00:19.691996 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.691979 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" Oct 30 20:00:19.692339 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.692002 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb" event={"ID":"b49b3d1e-1ee1-462a-80b2-7205b8063197","Type":"ContainerDied","Data":"e9fdb3a7ca7376c98b67a0684f5539ffe35cfc95a14e93bb6877727749fa7ff9"} Oct 30 20:00:19.692339 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.692059 4789 scope.go:122] "RemoveContainer" containerID="8195aa373f5ec32b26fadf9fb972a9123b122b94c34e2fa371ab92f0234e34d8" Oct 30 20:00:19.694972 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.694951 4789 generic.go:356] "Generic (PLEG): container finished" podID="78148758-1e4e-4117-b8d2-5ad6e83e28d5" containerID="f9baa79c42e20700704107c5a19e3717a6b57067b9ac411bacd6d3c62ce0f54b" exitCode=0 Oct 30 20:00:19.695091 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.695029 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" event={"ID":"78148758-1e4e-4117-b8d2-5ad6e83e28d5","Type":"ContainerDied","Data":"f9baa79c42e20700704107c5a19e3717a6b57067b9ac411bacd6d3c62ce0f54b"} Oct 30 20:00:19.696886 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.696866 4789 generic.go:356] "Generic (PLEG): container finished" podID="4a6787b8-a665-45b7-8b2f-52894a53f21d" containerID="844a6be0979dfa0ec64a31a5b420878cd5429cffcf211c24f63591bcee04fa18" exitCode=0 Oct 30 20:00:19.696983 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.696934 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" event={"ID":"4a6787b8-a665-45b7-8b2f-52894a53f21d","Type":"ContainerDied","Data":"844a6be0979dfa0ec64a31a5b420878cd5429cffcf211c24f63591bcee04fa18"} Oct 30 20:00:19.961983 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.961949 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:19.965789 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:19.965762 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-330025-29889840-d57nb"] Oct 30 20:00:20.863695 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.863668 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:20.886123 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.886096 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:20.941032 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.941007 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\") pod \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " Oct 30 20:00:20.941150 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.941097 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\") pod \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " Oct 30 20:00:20.941232 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.941155 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\" (UniqueName: \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\") pod \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\" (UID: \"78148758-1e4e-4117-b8d2-5ad6e83e28d5\") " Oct 30 20:00:20.941378 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.941356 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca" pod "78148758-1e4e-4117-b8d2-5ad6e83e28d5" (UID: "78148758-1e4e-4117-b8d2-5ad6e83e28d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.941482 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.941465 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.943097 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.943071 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca" pod "78148758-1e4e-4117-b8d2-5ad6e83e28d5" (UID: "78148758-1e4e-4117-b8d2-5ad6e83e28d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:20.943188 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:20.943148 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs" pod "78148758-1e4e-4117-b8d2-5ad6e83e28d5" (UID: "78148758-1e4e-4117-b8d2-5ad6e83e28d5"). InnerVolumeSpecName "kube-api-access-dbsvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.042784 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.042727 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\") pod \"4a6787b8-a665-45b7-8b2f-52894a53f21d\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " Oct 30 20:00:21.042915 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.042785 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\" (UniqueName: \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\") pod \"4a6787b8-a665-45b7-8b2f-52894a53f21d\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " Oct 30 20:00:21.042915 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.042848 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\") pod \"4a6787b8-a665-45b7-8b2f-52894a53f21d\" (UID: \"4a6787b8-a665-45b7-8b2f-52894a53f21d\") " Oct 30 20:00:21.043096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.043079 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbsvs\" (UniqueName: \"kubernetes.io/projected/78148758-1e4e-4117-b8d2-5ad6e83e28d5-kube-api-access-dbsvs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.043177 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.043101 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78148758-1e4e-4117-b8d2-5ad6e83e28d5-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.043242 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.043192 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca" pod "4a6787b8-a665-45b7-8b2f-52894a53f21d" (UID: "4a6787b8-a665-45b7-8b2f-52894a53f21d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.044738 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.044710 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca" pod "4a6787b8-a665-45b7-8b2f-52894a53f21d" (UID: "4a6787b8-a665-45b7-8b2f-52894a53f21d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:21.045026 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.045003 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf" pod "4a6787b8-a665-45b7-8b2f-52894a53f21d" (UID: "4a6787b8-a665-45b7-8b2f-52894a53f21d"). InnerVolumeSpecName "kube-api-access-f7pdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.144376 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.144342 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.144376 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.144369 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a6787b8-a665-45b7-8b2f-52894a53f21d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.144376 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.144380 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7pdf\" (UniqueName: \"kubernetes.io/projected/4a6787b8-a665-45b7-8b2f-52894a53f21d-kube-api-access-f7pdf\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.667255 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.667224 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" path="/var/lib/kubelet/pods/b49b3d1e-1ee1-462a-80b2-7205b8063197/volumes" Oct 30 20:00:21.711006 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.710981 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" Oct 30 20:00:21.711142 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.710998 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf06d798-kqzsj" event={"ID":"78148758-1e4e-4117-b8d2-5ad6e83e28d5","Type":"ContainerDied","Data":"ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a"} Oct 30 20:00:21.711142 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.711044 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed10acf3be2318f915139bcb995259e1b738c9407e5189864d4b9791f2e76e0a" Oct 30 20:00:21.712435 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.712420 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" Oct 30 20:00:21.712532 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.712464 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb91441-6f6mw" event={"ID":"4a6787b8-a665-45b7-8b2f-52894a53f21d","Type":"ContainerDied","Data":"a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64"} Oct 30 20:00:21.712532 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:21.712510 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a60299f1598e6e5942942b159c0868dc2ded204b2433713e858fb53525282d64" Oct 30 20:00:43.302613 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:43.302574 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4"] Oct 30 20:00:43.347312 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:43.347284 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-330025-sqdb4"] Oct 30 20:00:43.666635 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:43.666565 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c74e44-3a6e-4b0b-956c-e125cf1979a8" path="/var/lib/kubelet/pods/f5c74e44-3a6e-4b0b-956c-e125cf1979a8/volumes" Oct 30 20:00:47.795458 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:47.795403 4789 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pode809529b-0e7c-4a1d-b907-55043126357d"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pode809529b-0e7c-4a1d-b907-55043126357d] : Timed out while waiting for systemd to remove kubepods-burstable-pode809529b_0e7c_4a1d_b907_55043126357d.slice" Oct 30 20:00:47.795918 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:47.795458 4789 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pode809529b-0e7c-4a1d-b907-55043126357d] : unable to destroy cgroup paths for cgroup [kubepods burstable pode809529b-0e7c-4a1d-b907-55043126357d] : Timed out while waiting for systemd to remove kubepods-burstable-pode809529b_0e7c_4a1d_b907_55043126357d.slice" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" podUID="e809529b-0e7c-4a1d-b907-55043126357d" Oct 30 20:00:47.886650 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:47.886624 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv" Oct 30 20:00:48.049022 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:48.048956 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:48.061072 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:48.061044 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f77d05-29889840-vlclv"] Oct 30 20:00:48.195500 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:00:48.195455 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:00:49.667435 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:49.667394 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e809529b-0e7c-4a1d-b907-55043126357d" path="/var/lib/kubelet/pods/e809529b-0e7c-4a1d-b907-55043126357d/volumes" Oct 30 20:00:51.960470 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.960431 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z"] Oct 30 20:00:51.961040 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961019 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerName="main" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961042 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerName="main" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961070 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961077 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961087 4789 cpu_manager.go:414] "containerMap: removing container" podUID="4a6787b8-a665-45b7-8b2f-52894a53f21d" containerName="revoke-keys" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961095 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6787b8-a665-45b7-8b2f-52894a53f21d" containerName="revoke-keys" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961112 4789 cpu_manager.go:414] "containerMap: removing container" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961119 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" Oct 30 20:00:51.961132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961133 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961141 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961152 4789 cpu_manager.go:414] "containerMap: removing container" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961160 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961177 4789 cpu_manager.go:414] "containerMap: removing container" podUID="78148758-1e4e-4117-b8d2-5ad6e83e28d5" containerName="revoke-keys" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961184 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78148758-1e4e-4117-b8d2-5ad6e83e28d5" containerName="revoke-keys" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961335 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2db97b9-2af8-4f06-84d4-3cff4a5854ac" containerName="main" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961352 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961363 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6787b8-a665-45b7-8b2f-52894a53f21d" containerName="revoke-keys" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961373 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d69644f-e496-43f7-aa37-6264364d8078" containerName="main" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961384 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961394 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49b3d1e-1ee1-462a-80b2-7205b8063197" containerName="cleanup" Oct 30 20:00:51.961610 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.961405 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="78148758-1e4e-4117-b8d2-5ad6e83e28d5" containerName="revoke-keys" Oct 30 20:00:51.966279 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:51.966259 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.016893 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.016833 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:00:52.017055 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.016944 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:00:52.024466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.024440 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.024598 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.024493 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54gz\" (UniqueName: \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.024681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.024639 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.063167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.063147 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 20:00:52.063167 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.063159 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:52.065432 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.065412 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z"] Oct 30 20:00:52.125760 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.125733 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.125929 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.125790 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.125929 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.125831 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54gz\" (UniqueName: \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.126509 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.126492 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.128068 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.128051 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.184427 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.184400 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54gz\" (UniqueName: \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\") pod \"maas-api-revoke-sub-d228af00-k2v6z\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.279096 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.279041 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:52.659625 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.659602 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z"] Oct 30 20:00:52.661975 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:00:52.661950 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9456469c_3e83_4150_8785_9f63b7d17db7.slice/crio-39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e WatchSource:0}: Error finding container 39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e: Status 404 returned error can't find the container with id 39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e Oct 30 20:00:52.918998 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:52.918928 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" event={"ID":"9456469c-3e83-4150-8785-9f63b7d17db7","Type":"ContainerStarted","Data":"39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e"} Oct 30 20:00:53.926675 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:53.926609 4789 generic.go:356] "Generic (PLEG): container finished" podID="9456469c-3e83-4150-8785-9f63b7d17db7" containerID="a88eb50253fcfd01bbdd0041c47d332d5f9ab949a6a58d0ed4931a682976c293" exitCode=0 Oct 30 20:00:53.926675 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:53.926642 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" event={"ID":"9456469c-3e83-4150-8785-9f63b7d17db7","Type":"ContainerDied","Data":"a88eb50253fcfd01bbdd0041c47d332d5f9ab949a6a58d0ed4931a682976c293"} Oct 30 20:00:55.077283 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.077256 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:55.153171 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.153136 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\") pod \"9456469c-3e83-4150-8785-9f63b7d17db7\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " Oct 30 20:00:55.153355 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.153180 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\" (UniqueName: \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\") pod \"9456469c-3e83-4150-8785-9f63b7d17db7\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " Oct 30 20:00:55.153355 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.153267 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\") pod \"9456469c-3e83-4150-8785-9f63b7d17db7\" (UID: \"9456469c-3e83-4150-8785-9f63b7d17db7\") " Oct 30 20:00:55.153725 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.153694 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca" pod "9456469c-3e83-4150-8785-9f63b7d17db7" (UID: "9456469c-3e83-4150-8785-9f63b7d17db7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:55.155382 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.155356 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz" pod "9456469c-3e83-4150-8785-9f63b7d17db7" (UID: "9456469c-3e83-4150-8785-9f63b7d17db7"). InnerVolumeSpecName "kube-api-access-h54gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:55.155635 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.155618 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca" pod "9456469c-3e83-4150-8785-9f63b7d17db7" (UID: "9456469c-3e83-4150-8785-9f63b7d17db7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:55.254036 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.254004 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:55.254036 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.254036 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54gz\" (UniqueName: \"kubernetes.io/projected/9456469c-3e83-4150-8785-9f63b7d17db7-kube-api-access-h54gz\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:55.254239 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.254051 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9456469c-3e83-4150-8785-9f63b7d17db7-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:55.939387 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.939356 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" Oct 30 20:00:55.939541 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.939391 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d228af00-k2v6z" event={"ID":"9456469c-3e83-4150-8785-9f63b7d17db7","Type":"ContainerDied","Data":"39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e"} Oct 30 20:00:55.939541 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:00:55.939436 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f7395ba59e058643c8469a6bf3e36d64b22b59a25170a9eeb48280d4bd613e" Oct 30 20:01:04.665320 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:04.665292 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 20:01:04.698912 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:04.698881 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nlzxq_94f91f0e-7db9-426d-ac87-4c055d52abaa/console-operator/2.log" Oct 30 20:01:07.711679 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:07.711650 4789 scope.go:122] "RemoveContainer" containerID="0c8e5016a6f901d7d1362a3cc58343eb699d9eb6d8d2439b8b494f9cacbba127" Oct 30 20:01:07.722551 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:07.722531 4789 scope.go:122] "RemoveContainer" containerID="41098850b6197d181109a398222b63abe63823d32fc13f54ade6dbdf3b7bc19f" Oct 30 20:01:07.732112 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:07.732085 4789 scope.go:122] "RemoveContainer" containerID="dfd5b60edf83667dc4e8aa5a6c8fb181edafb864dc7c95e6eea9857c05031c60" Oct 30 20:01:18.195782 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:18.195744 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:01:18.747567 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.747535 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd"] Oct 30 20:01:18.747940 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.747927 4789 cpu_manager.go:414] "containerMap: removing container" podUID="9456469c-3e83-4150-8785-9f63b7d17db7" containerName="revoke-keys" Oct 30 20:01:18.747990 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.747941 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9456469c-3e83-4150-8785-9f63b7d17db7" containerName="revoke-keys" Oct 30 20:01:18.748045 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.748036 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="9456469c-3e83-4150-8785-9f63b7d17db7" containerName="revoke-keys" Oct 30 20:01:18.751328 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.751311 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.760042 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.760015 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfqc\" (UniqueName: \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.760164 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.760147 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.760273 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.760235 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.809385 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.809363 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd"] Oct 30 20:01:18.850875 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.850848 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:18.851015 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.850880 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:01:18.851015 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.850940 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:01:18.851015 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.850999 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 20:01:18.861241 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.861201 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.861518 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.861272 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.861518 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.861305 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfqc\" (UniqueName: \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.862093 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.862065 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.863934 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.863907 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:18.920247 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:18.920223 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfqc\" (UniqueName: \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\") pod \"maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:19.064378 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:19.064313 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:19.447798 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:19.447762 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd"] Oct 30 20:01:19.448810 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:01:19.448781 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc99cc0d2_856f_4271_8f79_ba4c06c6b12e.slice/crio-e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae WatchSource:0}: Error finding container e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae: Status 404 returned error can't find the container with id e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae Oct 30 20:01:20.096462 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:20.096381 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" event={"ID":"c99cc0d2-856f-4271-8f79-ba4c06c6b12e","Type":"ContainerStarted","Data":"8c6f75623c5c52f8e25c36574a0b05bab3bfd7e3bd2b0e41145c8ef49897eb90"} Oct 30 20:01:20.096462 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:20.096450 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" event={"ID":"c99cc0d2-856f-4271-8f79-ba4c06c6b12e","Type":"ContainerStarted","Data":"e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae"} Oct 30 20:01:20.208525 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:20.208470 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" podStartSLOduration=2.208457349 podStartE2EDuration="2.208457349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:20.206046533 +0000 UTC m=+4516.707957019" watchObservedRunningTime="2026-10-30 20:01:20.208457349 +0000 UTC m=+4516.710367835" Oct 30 20:01:21.104548 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:21.104509 4789 generic.go:356] "Generic (PLEG): container finished" podID="c99cc0d2-856f-4271-8f79-ba4c06c6b12e" containerID="8c6f75623c5c52f8e25c36574a0b05bab3bfd7e3bd2b0e41145c8ef49897eb90" exitCode=0 Oct 30 20:01:21.104945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:21.104591 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" event={"ID":"c99cc0d2-856f-4271-8f79-ba4c06c6b12e","Type":"ContainerDied","Data":"8c6f75623c5c52f8e25c36574a0b05bab3bfd7e3bd2b0e41145c8ef49897eb90"} Oct 30 20:01:22.255589 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.255564 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:22.293058 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.293028 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\") pod \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " Oct 30 20:01:22.293249 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.293229 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\" (UniqueName: \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\") pod \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " Oct 30 20:01:22.293329 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.293318 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\") pod \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\" (UID: \"c99cc0d2-856f-4271-8f79-ba4c06c6b12e\") " Oct 30 20:01:22.293722 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.293693 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca" pod "c99cc0d2-856f-4271-8f79-ba4c06c6b12e" (UID: "c99cc0d2-856f-4271-8f79-ba4c06c6b12e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:22.295108 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.295084 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca" pod "c99cc0d2-856f-4271-8f79-ba4c06c6b12e" (UID: "c99cc0d2-856f-4271-8f79-ba4c06c6b12e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:22.295325 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.295300 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc" pod "c99cc0d2-856f-4271-8f79-ba4c06c6b12e" (UID: "c99cc0d2-856f-4271-8f79-ba4c06c6b12e"). InnerVolumeSpecName "kube-api-access-bcfqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:22.394878 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.394799 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcfqc\" (UniqueName: \"kubernetes.io/projected/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-kube-api-access-bcfqc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.394878 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.394837 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.394878 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:22.394852 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c99cc0d2-856f-4271-8f79-ba4c06c6b12e-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:23.118643 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:23.118612 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" Oct 30 20:01:23.118813 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:23.118648 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd" event={"ID":"c99cc0d2-856f-4271-8f79-ba4c06c6b12e","Type":"ContainerDied","Data":"e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae"} Oct 30 20:01:23.118813 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:23.118694 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d24c4d0567531e41bff271505a95e5c0adea72d3a6487a626d8806130025ae" Oct 30 20:01:30.143654 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:30.143619 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 20:01:30.245273 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:30.245243 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 20:01:35.144548 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.144485 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" podUID="3cc44738-c830-4793-b8f9-2ab533a84f4e" containerName="payload-processing" containerID="cri-o://d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6" gracePeriod=25 Oct 30 20:01:35.245968 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.245913 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" podUID="f6b5c23e-4020-4358-8b01-183077b68965" containerName="payload-pre-processing" containerID="cri-o://c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe" gracePeriod=25 Oct 30 20:01:35.408575 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.408508 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 20:01:35.506693 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.506669 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\" (UniqueName: \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\") pod \"3cc44738-c830-4793-b8f9-2ab533a84f4e\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " Oct 30 20:01:35.506835 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.506774 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\") pod \"3cc44738-c830-4793-b8f9-2ab533a84f4e\" (UID: \"3cc44738-c830-4793-b8f9-2ab533a84f4e\") " Oct 30 20:01:35.507153 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.507131 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume" pod "3cc44738-c830-4793-b8f9-2ab533a84f4e" (UID: "3cc44738-c830-4793-b8f9-2ab533a84f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:35.508827 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.508806 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274" pod "3cc44738-c830-4793-b8f9-2ab533a84f4e" (UID: "3cc44738-c830-4793-b8f9-2ab533a84f4e"). InnerVolumeSpecName "kube-api-access-zq274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:35.509726 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.509709 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 20:01:35.607908 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.607879 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\") pod \"f6b5c23e-4020-4358-8b01-183077b68965\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " Oct 30 20:01:35.608101 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.607998 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\" (UniqueName: \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\") pod \"f6b5c23e-4020-4358-8b01-183077b68965\" (UID: \"f6b5c23e-4020-4358-8b01-183077b68965\") " Oct 30 20:01:35.608183 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.608162 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc44738-c830-4793-b8f9-2ab533a84f4e-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.608183 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.608178 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq274\" (UniqueName: \"kubernetes.io/projected/3cc44738-c830-4793-b8f9-2ab533a84f4e-kube-api-access-zq274\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.608370 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.608325 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume" pod "f6b5c23e-4020-4358-8b01-183077b68965" (UID: "f6b5c23e-4020-4358-8b01-183077b68965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:35.610052 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.610029 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s" pod "f6b5c23e-4020-4358-8b01-183077b68965" (UID: "f6b5c23e-4020-4358-8b01-183077b68965"). InnerVolumeSpecName "kube-api-access-62b2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:35.708724 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.708690 4789 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b5c23e-4020-4358-8b01-183077b68965-plugins-config-volume\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.708724 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:35.708719 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62b2s\" (UniqueName: \"kubernetes.io/projected/f6b5c23e-4020-4358-8b01-183077b68965-kube-api-access-62b2s\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:36.200738 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.200703 4789 generic.go:356] "Generic (PLEG): container finished" podID="f6b5c23e-4020-4358-8b01-183077b68965" containerID="c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe" exitCode=0 Oct 30 20:01:36.201204 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.200747 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" Oct 30 20:01:36.201204 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.200801 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" event={"ID":"f6b5c23e-4020-4358-8b01-183077b68965","Type":"ContainerDied","Data":"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe"} Oct 30 20:01:36.201204 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.200861 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92" event={"ID":"f6b5c23e-4020-4358-8b01-183077b68965","Type":"ContainerDied","Data":"b3ddccc821225b09c3945b0d0a34a045fb81c2538d2dd9004148b8bc602adbbf"} Oct 30 20:01:36.201204 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.200884 4789 scope.go:122] "RemoveContainer" containerID="c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe" Oct 30 20:01:36.202394 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.202365 4789 generic.go:356] "Generic (PLEG): container finished" podID="3cc44738-c830-4793-b8f9-2ab533a84f4e" containerID="d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6" exitCode=0 Oct 30 20:01:36.202502 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.202422 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" Oct 30 20:01:36.202502 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.202428 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" event={"ID":"3cc44738-c830-4793-b8f9-2ab533a84f4e","Type":"ContainerDied","Data":"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6"} Oct 30 20:01:36.202502 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.202468 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66" event={"ID":"3cc44738-c830-4793-b8f9-2ab533a84f4e","Type":"ContainerDied","Data":"7ebe4a17d5033ec87ab4f8041307de86e423f5a5ff8a5550efbedf2188865f35"} Oct 30 20:01:36.211044 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.211027 4789 scope.go:122] "RemoveContainer" containerID="c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe" Oct 30 20:01:36.211362 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:36.211343 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe\": container with ID starting with c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe not found: ID does not exist" containerID="c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe" Oct 30 20:01:36.211421 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.211370 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe"} err="failed to get container status \"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe\": rpc error: code = NotFound desc = could not find container \"c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe\": container with ID starting with c0ea4fac85be505cc69250b8da53b08f1d113560aaa44b79c64b55edd6f26afe not found: ID does not exist" Oct 30 20:01:36.211421 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.211387 4789 scope.go:122] "RemoveContainer" containerID="d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6" Oct 30 20:01:36.219784 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.219761 4789 scope.go:122] "RemoveContainer" containerID="d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6" Oct 30 20:01:36.220038 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:36.220019 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6\": container with ID starting with d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6 not found: ID does not exist" containerID="d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6" Oct 30 20:01:36.220114 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.220046 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6"} err="failed to get container status \"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6\": rpc error: code = NotFound desc = could not find container \"d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6\": container with ID starting with d9f0f89e7cd516de7854b91542df6a284f6c10ffa2ee3360d74de589812786a6 not found: ID does not exist" Oct 30 20:01:36.357561 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.357525 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 20:01:36.404000 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.403967 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d50b07-55bd6968bc-kgh92"] Oct 30 20:01:36.464959 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.464927 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 20:01:36.471874 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:36.471850 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d50b07-845bb6dbfb-29v66"] Oct 30 20:01:37.667602 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:37.667567 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc44738-c830-4793-b8f9-2ab533a84f4e" path="/var/lib/kubelet/pods/3cc44738-c830-4793-b8f9-2ab533a84f4e/volumes" Oct 30 20:01:37.668008 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:37.667993 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b5c23e-4020-4358-8b01-183077b68965" path="/var/lib/kubelet/pods/f6b5c23e-4020-4358-8b01-183077b68965/volumes" Oct 30 20:01:45.994561 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:45.994454 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd"] Oct 30 20:01:46.040501 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:46.040469 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d50b07-g4dbd"] Oct 30 20:01:47.668024 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:47.667989 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99cc0d2-856f-4271-8f79-ba4c06c6b12e" path="/var/lib/kubelet/pods/c99cc0d2-856f-4271-8f79-ba4c06c6b12e/volumes" Oct 30 20:01:48.195910 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:48.195871 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:01:53.069958 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.069913 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 20:01:53.070632 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.070231 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" containerID="cri-o://de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45" gracePeriod=30 Oct 30 20:01:53.953195 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953157 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk"] Oct 30 20:01:53.953580 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953566 4789 cpu_manager.go:414] "containerMap: removing container" podUID="f6b5c23e-4020-4358-8b01-183077b68965" containerName="payload-pre-processing" Oct 30 20:01:53.953627 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953582 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b5c23e-4020-4358-8b01-183077b68965" containerName="payload-pre-processing" Oct 30 20:01:53.953627 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953596 4789 cpu_manager.go:414] "containerMap: removing container" podUID="c99cc0d2-856f-4271-8f79-ba4c06c6b12e" containerName="revoke-keys" Oct 30 20:01:53.953627 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953602 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99cc0d2-856f-4271-8f79-ba4c06c6b12e" containerName="revoke-keys" Oct 30 20:01:53.953741 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953632 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3cc44738-c830-4793-b8f9-2ab533a84f4e" containerName="payload-processing" Oct 30 20:01:53.953741 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953639 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc44738-c830-4793-b8f9-2ab533a84f4e" containerName="payload-processing" Oct 30 20:01:53.953741 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953695 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc44738-c830-4793-b8f9-2ab533a84f4e" containerName="payload-processing" Oct 30 20:01:53.953741 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953704 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b5c23e-4020-4358-8b01-183077b68965" containerName="payload-pre-processing" Oct 30 20:01:53.953741 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.953711 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99cc0d2-856f-4271-8f79-ba4c06c6b12e" containerName="revoke-keys" Oct 30 20:01:53.957377 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:53.957358 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.012159 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.012111 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk"] Oct 30 20:01:54.055259 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.055189 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:01:54.055466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.055285 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:54.055466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.055309 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 20:01:54.055466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.055353 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:01:54.079964 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.079934 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.080400 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.079970 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.080400 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.080080 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrdm\" (UniqueName: \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.181470 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.181429 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.181470 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.181471 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.181688 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.181515 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrdm\" (UniqueName: \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.182183 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.182151 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.183822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.183799 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.244197 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.244120 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrdm\" (UniqueName: \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\") pod \"maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.270006 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.269968 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:54.646713 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:54.646682 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk"] Oct 30 20:01:54.649540 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:01:54.649509 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd27e7f_0f73_46a3_9c4c_b463217cadba.slice/crio-7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840 WatchSource:0}: Error finding container 7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840: Status 404 returned error can't find the container with id 7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840 Oct 30 20:01:55.321767 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:55.321724 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" event={"ID":"fdd27e7f-0f73-46a3-9c4c-b463217cadba","Type":"ContainerStarted","Data":"36da975d1df3e37e3ade85b6287a09a8a372ea72ea29516784cc22c9ac091ccd"} Oct 30 20:01:55.322294 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:55.321782 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" event={"ID":"fdd27e7f-0f73-46a3-9c4c-b463217cadba","Type":"ContainerStarted","Data":"7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840"} Oct 30 20:01:55.435681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:55.435623 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" podStartSLOduration=2.435609449 podStartE2EDuration="2.435609449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:55.434491657 +0000 UTC m=+4551.936402144" watchObservedRunningTime="2026-10-30 20:01:55.435609449 +0000 UTC m=+4551.937519935" Oct 30 20:01:56.329520 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:56.329480 4789 generic.go:356] "Generic (PLEG): container finished" podID="fdd27e7f-0f73-46a3-9c4c-b463217cadba" containerID="36da975d1df3e37e3ade85b6287a09a8a372ea72ea29516784cc22c9ac091ccd" exitCode=0 Oct 30 20:01:56.329931 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:56.329562 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" event={"ID":"fdd27e7f-0f73-46a3-9c4c-b463217cadba","Type":"ContainerDied","Data":"36da975d1df3e37e3ade85b6287a09a8a372ea72ea29516784cc22c9ac091ccd"} Oct 30 20:01:57.484962 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.484936 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:57.614533 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.614452 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\" (UniqueName: \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\") pod \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " Oct 30 20:01:57.614681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.614549 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\") pod \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " Oct 30 20:01:57.614749 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.614691 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\") pod \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\" (UID: \"fdd27e7f-0f73-46a3-9c4c-b463217cadba\") " Oct 30 20:01:57.615172 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.615139 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca" pod "fdd27e7f-0f73-46a3-9c4c-b463217cadba" (UID: "fdd27e7f-0f73-46a3-9c4c-b463217cadba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:57.616844 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.616816 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca" pod "fdd27e7f-0f73-46a3-9c4c-b463217cadba" (UID: "fdd27e7f-0f73-46a3-9c4c-b463217cadba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:57.616967 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.616817 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm" pod "fdd27e7f-0f73-46a3-9c4c-b463217cadba" (UID: "fdd27e7f-0f73-46a3-9c4c-b463217cadba"). InnerVolumeSpecName "kube-api-access-sdrdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:57.716306 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.716267 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:57.716306 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.716299 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrdm\" (UniqueName: \"kubernetes.io/projected/fdd27e7f-0f73-46a3-9c4c-b463217cadba-kube-api-access-sdrdm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:57.716517 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:57.716313 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdd27e7f-0f73-46a3-9c4c-b463217cadba-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.324573 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.324546 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 20:01:58.346719 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.346642 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" Oct 30 20:01:58.346719 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.346651 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk" event={"ID":"fdd27e7f-0f73-46a3-9c4c-b463217cadba","Type":"ContainerDied","Data":"7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840"} Oct 30 20:01:58.346719 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.346699 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a39e750e85755d11b0f98c8cce373c7ff6a165848ed053d360178b337ff6840" Oct 30 20:01:58.348706 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.348677 4789 generic.go:356] "Generic (PLEG): container finished" podID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerID="de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45" exitCode=0 Oct 30 20:01:58.348895 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.348753 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" Oct 30 20:01:58.348895 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.348770 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerDied","Data":"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45"} Oct 30 20:01:58.348895 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.348813 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl" event={"ID":"551c106f-e5b9-47c2-a48d-78eec02a7561","Type":"ContainerDied","Data":"23f78ad36fafbbef6c9984782d2f61d2f90ccae43b0b4c9d409d37acd6fe7003"} Oct 30 20:01:58.348895 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.348837 4789 scope.go:122] "RemoveContainer" containerID="de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45" Oct 30 20:01:58.358677 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.358654 4789 scope.go:122] "RemoveContainer" containerID="3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7" Oct 30 20:01:58.367061 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.367043 4789 scope.go:122] "RemoveContainer" containerID="de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45" Oct 30 20:01:58.367339 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:58.367320 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45\": container with ID starting with de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45 not found: ID does not exist" containerID="de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45" Oct 30 20:01:58.367391 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.367347 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45"} err="failed to get container status \"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45\": rpc error: code = NotFound desc = could not find container \"de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45\": container with ID starting with de44478788c54c34d0ed3da16b8d2ddec0c25196f9fc0c6bb3227afaccab8f45 not found: ID does not exist" Oct 30 20:01:58.367391 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.367363 4789 scope.go:122] "RemoveContainer" containerID="3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7" Oct 30 20:01:58.367618 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:01:58.367598 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7\": container with ID starting with 3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7 not found: ID does not exist" containerID="3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7" Oct 30 20:01:58.367664 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.367624 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7"} err="failed to get container status \"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7\": rpc error: code = NotFound desc = could not find container \"3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7\": container with ID starting with 3ed9a0892217d193255de82d5fab5a1ed6905f9d3aa6fb3744eaecb09927b2c7 not found: ID does not exist" Oct 30 20:01:58.421592 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421554 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421728 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421614 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421728 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421643 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421728 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421683 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421728 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421707 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421965 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421741 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421965 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421787 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421965 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421813 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.421965 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421841 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\") pod \"551c106f-e5b9-47c2-a48d-78eec02a7561\" (UID: \"551c106f-e5b9-47c2-a48d-78eec02a7561\") " Oct 30 20:01:58.422201 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.421977 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:58.422201 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422047 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:58.422365 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422274 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:58.422365 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422277 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.422365 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422305 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.422365 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422315 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:58.422590 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.422374 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:58.423747 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.423725 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:58.423929 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.423907 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "kube-api-access-c44kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:58.423974 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.423910 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:58.423974 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.423951 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo" pod "551c106f-e5b9-47c2-a48d-78eec02a7561" (UID: "551c106f-e5b9-47c2-a48d-78eec02a7561"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:58.522885 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522852 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c44kv\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-kube-api-access-c44kv\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.522885 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522883 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.522885 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522894 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.523479 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522905 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.523479 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522920 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551c106f-e5b9-47c2-a48d-78eec02a7561-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.523479 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522934 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.523479 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.522948 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551c106f-e5b9-47c2-a48d-78eec02a7561-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:58.764376 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.764341 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 20:01:58.772146 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:58.772117 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d50b07-openshift-default-5df9b66bf7-8r4wl"] Oct 30 20:01:59.666703 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:01:59.666665 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" path="/var/lib/kubelet/pods/551c106f-e5b9-47c2-a48d-78eec02a7561/volumes" Oct 30 20:02:07.780230 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:07.780174 4789 scope.go:122] "RemoveContainer" containerID="ec4fb5a165b49bd7f1bb21aa4ade2cfa69c13c744f3f2b42ca224fdce1e1a0a9" Oct 30 20:02:07.789454 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:07.789432 4789 scope.go:122] "RemoveContainer" containerID="376a1c4184afdd9a60ec2ddf5ab977e239e66e123052c7dd6a80de764a787bb8" Oct 30 20:02:15.425091 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:15.425048 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk"] Oct 30 20:02:15.471200 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:15.471167 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f77d05-h67jk"] Oct 30 20:02:15.666925 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:15.666889 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd27e7f-0f73-46a3-9c4c-b463217cadba" path="/var/lib/kubelet/pods/fdd27e7f-0f73-46a3-9c4c-b463217cadba/volumes" Oct 30 20:02:18.196255 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:18.196200 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:02:23.167284 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:23.167248 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 20:02:23.167785 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:23.167533 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" containerID="cri-o://a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea" gracePeriod=30 Oct 30 20:02:28.433609 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.433550 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 20:02:28.489829 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.489800 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490009 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.489837 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490009 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.489912 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490009 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.489930 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490009 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.489973 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490009 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490003 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490337 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490043 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490337 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490079 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490337 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490165 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\") pod \"eb8e50bf-801f-4219-b2db-93f41a3c4784\" (UID: \"eb8e50bf-801f-4219-b2db-93f41a3c4784\") " Oct 30 20:02:28.490337 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490304 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:28.490566 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490446 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:28.490566 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490526 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:28.490681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490635 4789 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-credential-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.490681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490640 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:28.490681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490659 4789 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.490681 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490675 4789 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-workload-socket\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.490893 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.490841 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:28.492222 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.492192 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:28.492313 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.492227 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:28.492494 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.492471 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:28.492607 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.492509 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p" pod "eb8e50bf-801f-4219-b2db-93f41a3c4784" (UID: "eb8e50bf-801f-4219-b2db-93f41a3c4784"). InnerVolumeSpecName "kube-api-access-fdc7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:28.529145 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.529113 4789 generic.go:356] "Generic (PLEG): container finished" podID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerID="a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea" exitCode=0 Oct 30 20:02:28.529312 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.529167 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" Oct 30 20:02:28.529312 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.529200 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerDied","Data":"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea"} Oct 30 20:02:28.529312 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.529282 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh" event={"ID":"eb8e50bf-801f-4219-b2db-93f41a3c4784","Type":"ContainerDied","Data":"b5032e650a68cf10af7d33de9864d92c73248c320c0e695eda26790bf386b807"} Oct 30 20:02:28.529504 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.529316 4789 scope.go:122] "RemoveContainer" containerID="a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea" Oct 30 20:02:28.539261 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.539244 4789 scope.go:122] "RemoveContainer" containerID="9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd" Oct 30 20:02:28.548040 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.548022 4789 scope.go:122] "RemoveContainer" containerID="a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea" Oct 30 20:02:28.548302 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:28.548283 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea\": container with ID starting with a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea not found: ID does not exist" containerID="a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea" Oct 30 20:02:28.548383 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.548316 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea"} err="failed to get container status \"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea\": rpc error: code = NotFound desc = could not find container \"a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea\": container with ID starting with a8390ba8478285584c67a74f42a8563e7c1891c09d790427af21dee5ebcf0aea not found: ID does not exist" Oct 30 20:02:28.548383 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.548340 4789 scope.go:122] "RemoveContainer" containerID="9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd" Oct 30 20:02:28.548598 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:28.548580 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd\": container with ID starting with 9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd not found: ID does not exist" containerID="9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd" Oct 30 20:02:28.548640 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.548605 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd"} err="failed to get container status \"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd\": rpc error: code = NotFound desc = could not find container \"9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd\": container with ID starting with 9283fbfd950c835b0520515a53e122d2527a5d774570c40cb4a779730947d5bd not found: ID does not exist" Oct 30 20:02:28.591448 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591423 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc7p\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-kube-api-access-fdc7p\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.591448 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591447 4789 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8e50bf-801f-4219-b2db-93f41a3c4784-istiod-ca-cert\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.591599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591464 4789 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-token\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.591599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591473 4789 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-podinfo\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.591599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591482 4789 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-data\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.591599 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.591490 4789 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8e50bf-801f-4219-b2db-93f41a3c4784-istio-envoy\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:28.644218 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.644187 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 20:02:28.646486 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.646457 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 20:02:28.654948 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.654923 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 20:02:28.703102 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.703070 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 20:02:28.708728 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:28.708703 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f77d05-openshift-default-7f9f46f7d9-swsqh"] Oct 30 20:02:29.668125 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:29.668091 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" path="/var/lib/kubelet/pods/eb8e50bf-801f-4219-b2db-93f41a3c4784/volumes" Oct 30 20:02:38.564503 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564469 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj"] Oct 30 20:02:38.564891 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564842 4789 cpu_manager.go:414] "containerMap: removing container" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:02:38.564891 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564855 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:02:38.564891 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564869 4789 cpu_manager.go:414] "containerMap: removing container" podUID="fdd27e7f-0f73-46a3-9c4c-b463217cadba" containerName="revoke-keys" Oct 30 20:02:38.564891 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564875 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd27e7f-0f73-46a3-9c4c-b463217cadba" containerName="revoke-keys" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564895 4789 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564902 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564913 4789 cpu_manager.go:414] "containerMap: removing container" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564919 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564981 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564990 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.564997 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8e50bf-801f-4219-b2db-93f41a3c4784" containerName="istio-proxy" Oct 30 20:02:38.565035 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.565007 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd27e7f-0f73-46a3-9c4c-b463217cadba" containerName="revoke-keys" Oct 30 20:02:38.569819 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.569796 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.624634 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.624602 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj"] Oct 30 20:02:38.667736 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.667708 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:02:38.667872 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.667737 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:02:38.667872 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.667747 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:38.667872 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.667803 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8jtx\"" Oct 30 20:02:38.679298 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.679280 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkf8\" (UniqueName: \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.679366 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.679329 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.679513 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.679494 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.780300 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.780273 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkf8\" (UniqueName: \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.780454 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.780330 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.780454 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.780413 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.781031 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.781002 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.782803 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.782780 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.838857 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.838800 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkf8\" (UniqueName: \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\") pod \"maas-api-revoke-sub-201e45b9-wzmqj\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:38.882276 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:38.882253 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:39.250439 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:39.250267 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj"] Oct 30 20:02:39.252750 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:02:39.252719 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7926ea8_1e00_4623_a82d_4606bbb2176d.slice/crio-16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12 WatchSource:0}: Error finding container 16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12: Status 404 returned error can't find the container with id 16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12 Oct 30 20:02:39.600148 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:39.600064 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" event={"ID":"a7926ea8-1e00-4623-a82d-4606bbb2176d","Type":"ContainerStarted","Data":"35a3836c19dcb9afa530cab9df1b9ce0a75ffd82d27ff6baa211c29a4362ad8b"} Oct 30 20:02:39.600148 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:39.600126 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" event={"ID":"a7926ea8-1e00-4623-a82d-4606bbb2176d","Type":"ContainerStarted","Data":"16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12"} Oct 30 20:02:39.712656 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:39.712599 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" podStartSLOduration=1.712586587 podStartE2EDuration="1.712586587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:39.710403729 +0000 UTC m=+4596.212314217" watchObservedRunningTime="2026-10-30 20:02:39.712586587 +0000 UTC m=+4596.214497072" Oct 30 20:02:40.607844 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:40.607800 4789 generic.go:356] "Generic (PLEG): container finished" podID="a7926ea8-1e00-4623-a82d-4606bbb2176d" containerID="35a3836c19dcb9afa530cab9df1b9ce0a75ffd82d27ff6baa211c29a4362ad8b" exitCode=0 Oct 30 20:02:40.608269 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:40.607874 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" event={"ID":"a7926ea8-1e00-4623-a82d-4606bbb2176d","Type":"ContainerDied","Data":"35a3836c19dcb9afa530cab9df1b9ce0a75ffd82d27ff6baa211c29a4362ad8b"} Oct 30 20:02:41.752141 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.752108 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:41.909024 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.908919 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\") pod \"a7926ea8-1e00-4623-a82d-4606bbb2176d\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " Oct 30 20:02:41.909024 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.908986 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\") pod \"a7926ea8-1e00-4623-a82d-4606bbb2176d\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " Oct 30 20:02:41.909024 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.909029 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\" (UniqueName: \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\") pod \"a7926ea8-1e00-4623-a82d-4606bbb2176d\" (UID: \"a7926ea8-1e00-4623-a82d-4606bbb2176d\") " Oct 30 20:02:41.909365 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.909302 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca" pod "a7926ea8-1e00-4623-a82d-4606bbb2176d" (UID: "a7926ea8-1e00-4623-a82d-4606bbb2176d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:41.911188 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.911154 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca" pod "a7926ea8-1e00-4623-a82d-4606bbb2176d" (UID: "a7926ea8-1e00-4623-a82d-4606bbb2176d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:41.911381 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:41.911202 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8" pod "a7926ea8-1e00-4623-a82d-4606bbb2176d" (UID: "a7926ea8-1e00-4623-a82d-4606bbb2176d"). InnerVolumeSpecName "kube-api-access-wmkf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:42.010641 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.010609 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-openshift-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:42.010641 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.010638 4789 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7926ea8-1e00-4623-a82d-4606bbb2176d-maas-api-xks-ca\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:42.010641 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.010647 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkf8\" (UniqueName: \"kubernetes.io/projected/a7926ea8-1e00-4623-a82d-4606bbb2176d-kube-api-access-wmkf8\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:42.620978 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.620950 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" Oct 30 20:02:42.621131 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.620948 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-201e45b9-wzmqj" event={"ID":"a7926ea8-1e00-4623-a82d-4606bbb2176d","Type":"ContainerDied","Data":"16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12"} Oct 30 20:02:42.621131 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:42.621078 4789 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16213dbe9aa28c7739be7e1a86bb5d15ab568a5a452811156d9e77ee582dca12" Oct 30 20:02:43.664516 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.664464 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" containerID="cri-o://99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d" gracePeriod=45 Oct 30 20:02:43.666141 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.666114 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" containerID="cri-o://ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a" gracePeriod=45 Oct 30 20:02:43.674968 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.674943 4789 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" podUID="29dbf484-abcb-4246-8e97-9784359be27c" containerName="main" containerID="cri-o://fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718" gracePeriod=45 Oct 30 20:02:43.810537 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.810492 4789 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": dial tcp 10.132.0.58:8000: connect: connection refused" Oct 30 20:02:43.934169 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.934143 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 20:02:43.954401 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.954376 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 20:02:43.964107 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:43.964087 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 20:02:44.027201 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027173 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027358 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027227 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\" (UniqueName: \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027358 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027272 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027465 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027382 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027465 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027426 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027587 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027465 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\") pod \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\" (UID: \"a8575aa9-12e3-474c-b85f-e14817c4d6ac\") " Oct 30 20:02:44.027587 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027510 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.027740 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027705 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.027827 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027744 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.027827 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.027781 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.029677 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.029645 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "kube-api-access-4b289". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:44.029789 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.029673 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:44.029866 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.029840 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm" pod "a8575aa9-12e3-474c-b85f-e14817c4d6ac" (UID: "a8575aa9-12e3-474c-b85f-e14817c4d6ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.128175 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128140 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\" (UniqueName: \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128175 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128180 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128197 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128231 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128276 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128300 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\" (UniqueName: \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128353 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128466 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128416 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128481 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128477 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128508 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128560 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128577 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128592 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\") pod \"3a27d90e-3c13-438b-a204-74dc08b39517\" (UID: \"3a27d90e-3c13-438b-a204-74dc08b39517\") " Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128626 4789 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\" (UniqueName: \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\") pod \"29dbf484-abcb-4246-8e97-9784359be27c\" (UID: \"29dbf484-abcb-4246-8e97-9784359be27c\") " Oct 30 20:02:44.128822 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128805 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128961 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.128968 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129004 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b289\" (UniqueName: \"kubernetes.io/projected/a8575aa9-12e3-474c-b85f-e14817c4d6ac-kube-api-access-4b289\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129024 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129039 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129052 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129066 4789 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-model-cache\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129082 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129096 4789 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tmp-dir\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129135 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8575aa9-12e3-474c-b85f-e14817c4d6ac-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.129295 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.129168 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.130596 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.130565 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "kube-api-access-xcprk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:44.130716 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.130656 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "kube-api-access-22jxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:44.130782 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.130765 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.130967 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.130939 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs" pod "3a27d90e-3c13-438b-a204-74dc08b39517" (UID: "3a27d90e-3c13-438b-a204-74dc08b39517"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:44.131080 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.131000 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:44.131132 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.131121 4789 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm" pod "29dbf484-abcb-4246-8e97-9784359be27c" (UID: "29dbf484-abcb-4246-8e97-9784359be27c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:44.229913 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229876 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a27d90e-3c13-438b-a204-74dc08b39517-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.229913 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229915 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229929 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dbf484-abcb-4246-8e97-9784359be27c-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229942 4789 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-home\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229958 4789 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dbf484-abcb-4246-8e97-9784359be27c-tls-certs\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229972 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jxc\" (UniqueName: \"kubernetes.io/projected/3a27d90e-3c13-438b-a204-74dc08b39517-kube-api-access-22jxc\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.229988 4789 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a27d90e-3c13-438b-a204-74dc08b39517-dshm\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.230173 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.230002 4789 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcprk\" (UniqueName: \"kubernetes.io/projected/29dbf484-abcb-4246-8e97-9784359be27c-kube-api-access-xcprk\") on node \"ip-10-0-136-46.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:44.634438 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.634361 4789 generic.go:356] "Generic (PLEG): container finished" podID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerID="ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a" exitCode=0 Oct 30 20:02:44.634438 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.634417 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" Oct 30 20:02:44.634676 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.634438 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" event={"ID":"a8575aa9-12e3-474c-b85f-e14817c4d6ac","Type":"ContainerDied","Data":"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a"} Oct 30 20:02:44.634676 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.634478 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp" event={"ID":"a8575aa9-12e3-474c-b85f-e14817c4d6ac","Type":"ContainerDied","Data":"45141f25906f5da2779406ed6146bfd9728c861b37593cc682c980f06336f457"} Oct 30 20:02:44.634676 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.634501 4789 scope.go:122] "RemoveContainer" containerID="ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a" Oct 30 20:02:44.635859 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.635832 4789 generic.go:356] "Generic (PLEG): container finished" podID="3a27d90e-3c13-438b-a204-74dc08b39517" containerID="99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d" exitCode=0 Oct 30 20:02:44.635970 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.635896 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" Oct 30 20:02:44.635970 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.635914 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" event={"ID":"3a27d90e-3c13-438b-a204-74dc08b39517","Type":"ContainerDied","Data":"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d"} Oct 30 20:02:44.635970 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.635953 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72" event={"ID":"3a27d90e-3c13-438b-a204-74dc08b39517","Type":"ContainerDied","Data":"6d8f19751efe5b12ca4793714b7ea8dd214c6bf085bbe211b7747a595c1ab21f"} Oct 30 20:02:44.637742 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.637717 4789 generic.go:356] "Generic (PLEG): container finished" podID="29dbf484-abcb-4246-8e97-9784359be27c" containerID="fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718" exitCode=0 Oct 30 20:02:44.637869 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.637765 4789 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" Oct 30 20:02:44.637869 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.637795 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" event={"ID":"29dbf484-abcb-4246-8e97-9784359be27c","Type":"ContainerDied","Data":"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718"} Oct 30 20:02:44.637869 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.637840 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2" event={"ID":"29dbf484-abcb-4246-8e97-9784359be27c","Type":"ContainerDied","Data":"333dca9bcf451e244c9e60511034e17570e6fbdf8b5d90ba429880b849d250e6"} Oct 30 20:02:44.646874 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.646860 4789 scope.go:122] "RemoveContainer" containerID="ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a" Oct 30 20:02:44.647127 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:44.647105 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a\": container with ID starting with ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a not found: ID does not exist" containerID="ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a" Oct 30 20:02:44.647190 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.647134 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a"} err="failed to get container status \"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a\": rpc error: code = NotFound desc = could not find container \"ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a\": container with ID starting with ec204d821f28be6d706d86db53a31e0aaf88a14fa3982afadb020d7a66e5de6a not found: ID does not exist" Oct 30 20:02:44.647190 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.647151 4789 scope.go:122] "RemoveContainer" containerID="99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d" Oct 30 20:02:44.655139 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.655117 4789 scope.go:122] "RemoveContainer" containerID="99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d" Oct 30 20:02:44.655407 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:44.655390 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d\": container with ID starting with 99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d not found: ID does not exist" containerID="99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d" Oct 30 20:02:44.655485 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.655414 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d"} err="failed to get container status \"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d\": rpc error: code = NotFound desc = could not find container \"99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d\": container with ID starting with 99fba034f6da8be8af8f688844d326571af89fc303987685a8e545a5e4f8a67d not found: ID does not exist" Oct 30 20:02:44.655485 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.655431 4789 scope.go:122] "RemoveContainer" containerID="fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718" Oct 30 20:02:44.663243 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.663222 4789 scope.go:122] "RemoveContainer" containerID="fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718" Oct 30 20:02:44.663516 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:44.663491 4789 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718\": container with ID starting with fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718 not found: ID does not exist" containerID="fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718" Oct 30 20:02:44.663584 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.663513 4789 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718"} err="failed to get container status \"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718\": rpc error: code = NotFound desc = could not find container \"fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718\": container with ID starting with fdfab6c467c26c1012cd59bd9601ae8b9bac5a69c17411b4893cb60601ef1718 not found: ID does not exist" Oct 30 20:02:44.790227 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.790192 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 20:02:44.836246 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.836223 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/premium-w4-1323a7-kserve-8445798475-hcpdp"] Oct 30 20:02:44.940953 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.940929 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 20:02:44.944980 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:44.944961 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/distinct-w4-1323a7-kserve-6fb6766dfd-xgs72"] Oct 30 20:02:45.050296 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:45.050274 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 20:02:45.084576 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:45.084554 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1323a7/simulator-w4-1323a7-kserve-6ddc948ccd-tdhk2"] Oct 30 20:02:45.666119 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:45.666088 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29dbf484-abcb-4246-8e97-9784359be27c" path="/var/lib/kubelet/pods/29dbf484-abcb-4246-8e97-9784359be27c/volumes" Oct 30 20:02:45.666507 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:45.666492 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" path="/var/lib/kubelet/pods/3a27d90e-3c13-438b-a204-74dc08b39517/volumes" Oct 30 20:02:45.666871 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:02:45.666859 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" path="/var/lib/kubelet/pods/a8575aa9-12e3-474c-b85f-e14817c4d6ac/volumes" Oct 30 20:02:48.194226 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:02:48.194177 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:03:05.819914 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:05.819878 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6vfk4_36e0b317-a46f-48dc-9c68-501c0d36bef9/manager/0.log" Oct 30 20:03:07.193633 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.193573 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kkqxp_05644db3-a8f4-4fad-8b6a-0713ac0c8f01/manager/0.log" Oct 30 20:03:07.696861 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.696833 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rxf5s_b4d866b2-7844-4f4a-ba79-31d75d4d8e76/manager/0.log" Oct 30 20:03:07.833429 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.832057 4789 scope.go:122] "RemoveContainer" containerID="141eb5480a9ad118625ffb7f5100cfbce541c3d7356af435e3fd065b6bdf36a0" Oct 30 20:03:07.847950 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.847801 4789 scope.go:122] "RemoveContainer" containerID="1d2c3a60256e621821db2a2e0a9c5f4349fe6cd591dfad8fbcb00a4f21a79363" Oct 30 20:03:07.857191 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.857169 4789 scope.go:122] "RemoveContainer" containerID="5ad5184e992ef65567cc561286b5eba31accb2f13118613c278391a13b849e71" Oct 30 20:03:07.865028 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:07.865002 4789 scope.go:122] "RemoveContainer" containerID="de986c7ddcbcb8a2d19f17f6167728de18b32ec98c37631aa2a8937d38806cbc" Oct 30 20:03:09.449597 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:09.449554 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:03:10.505521 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:10.505490 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:03:18.196286 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:03:18.196197 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:03:23.609085 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:23.609055 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:03:24.632407 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:24.632374 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:03:26.370503 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:26.370474 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-201e45b9-wzmqj_a7926ea8-1e00-4623-a82d-4606bbb2176d/revoke-keys/0.log" Oct 30 20:03:26.844392 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:26.844358 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4e6a7660-nfhrw_8d6be7de-7a61-4f00-97ca-e157806ccaef/revoke-keys/0.log" Oct 30 20:03:27.106409 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:27.106348 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5bbaa2a2-jjrz8_26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0/revoke-keys/0.log" Oct 30 20:03:27.358669 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:27.358600 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aed74471-qvs6z_f6bb431c-f448-49a9-9557-0a98da55fbb9/revoke-keys/0.log" Oct 30 20:03:27.598540 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:27.598517 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bdb91441-6f6mw_4a6787b8-a665-45b7-8b2f-52894a53f21d/revoke-keys/0.log" Oct 30 20:03:27.831797 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:27.831771 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bf06d798-kqzsj_78148758-1e4e-4117-b8d2-5ad6e83e28d5/revoke-keys/0.log" Oct 30 20:03:28.062923 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:28.062899 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cd5c480a-whv9j_c0bf77b8-a6c1-44aa-9003-2212327d0a04/revoke-keys/0.log" Oct 30 20:03:28.291672 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:28.291646 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d228af00-k2v6z_9456469c-3e83-4150-8785-9f63b7d17db7/revoke-keys/0.log" Oct 30 20:03:28.509090 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:28.509064 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d5cabe1d-mhjv2_f6d256cb-f374-476d-808a-8d2bb9c8b794/revoke-keys/0.log" Oct 30 20:03:28.725440 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:28.725417 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e2ab9f3a-rrp7q_bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc/revoke-keys/0.log" Oct 30 20:03:31.139888 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:31.139844 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6vfk4_36e0b317-a46f-48dc-9c68-501c0d36bef9/manager/0.log" Oct 30 20:03:32.331014 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:32.330971 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kkqxp_05644db3-a8f4-4fad-8b6a-0713ac0c8f01/manager/0.log" Oct 30 20:03:32.827100 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:32.827069 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rxf5s_b4d866b2-7844-4f4a-ba79-31d75d4d8e76/manager/0.log" Oct 30 20:03:34.521738 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:34.521709 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df7c6869c-nrlk4_4cae15c3-20da-4a51-8b5c-3386568b4d72/kube-auth-proxy/0.log" Oct 30 20:03:35.032267 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:35.032241 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp_45bae6ce-241c-4f8d-b724-6ea46abc05e6/istio-proxy/0.log" Oct 30 20:03:35.758835 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:35.758802 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5978b965c7-kbthb_0605710c-1a4a-47f2-a7bc-4079a97e9db4/router/0.log" Oct 30 20:03:48.199161 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:03:48.199109 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:03:49.854403 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854372 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j68lw/must-gather-4dlfv"] Oct 30 20:03:49.854773 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854719 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" Oct 30 20:03:49.854773 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854730 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" Oct 30 20:03:49.854773 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854747 4789 cpu_manager.go:414] "containerMap: removing container" podUID="a7926ea8-1e00-4623-a82d-4606bbb2176d" containerName="revoke-keys" Oct 30 20:03:49.854773 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854752 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7926ea8-1e00-4623-a82d-4606bbb2176d" containerName="revoke-keys" Oct 30 20:03:49.854773 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854773 4789 cpu_manager.go:414] "containerMap: removing container" podUID="29dbf484-abcb-4246-8e97-9784359be27c" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854778 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29dbf484-abcb-4246-8e97-9784359be27c" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854784 4789 cpu_manager.go:414] "containerMap: removing container" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854789 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854794 4789 cpu_manager.go:414] "containerMap: removing container" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854799 4789 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854854 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="551c106f-e5b9-47c2-a48d-78eec02a7561" containerName="istio-proxy" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854863 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7926ea8-1e00-4623-a82d-4606bbb2176d" containerName="revoke-keys" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854873 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="29dbf484-abcb-4246-8e97-9784359be27c" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854877 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a27d90e-3c13-438b-a204-74dc08b39517" containerName="main" Oct 30 20:03:49.854945 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.854885 4789 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8575aa9-12e3-474c-b85f-e14817c4d6ac" containerName="main" Oct 30 20:03:49.857898 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.857882 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:49.915177 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.915150 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j68lw/must-gather-4dlfv"] Oct 30 20:03:49.927995 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.927970 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659rp\" (UniqueName: \"kubernetes.io/projected/f638db36-367a-4e15-b6bf-330132786526-kube-api-access-659rp\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:49.928102 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.928082 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f638db36-367a-4e15-b6bf-330132786526-must-gather-output\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:49.958757 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.958737 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j68lw\"/\"openshift-service-ca.crt\"" Oct 30 20:03:49.959025 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.959007 4789 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j68lw\"/\"default-dockercfg-plqkf\"" Oct 30 20:03:49.959111 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:49.959014 4789 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j68lw\"/\"kube-root-ca.crt\"" Oct 30 20:03:50.028907 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.028878 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f638db36-367a-4e15-b6bf-330132786526-must-gather-output\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:50.029050 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.028949 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659rp\" (UniqueName: \"kubernetes.io/projected/f638db36-367a-4e15-b6bf-330132786526-kube-api-access-659rp\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:50.029205 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.029183 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f638db36-367a-4e15-b6bf-330132786526-must-gather-output\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:50.095910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.095883 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659rp\" (UniqueName: \"kubernetes.io/projected/f638db36-367a-4e15-b6bf-330132786526-kube-api-access-659rp\") pod \"must-gather-4dlfv\" (UID: \"f638db36-367a-4e15-b6bf-330132786526\") " pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:50.172103 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.172051 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j68lw/must-gather-4dlfv" Oct 30 20:03:50.559046 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:03:50.559018 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf638db36_367a_4e15_b6bf_330132786526.slice/crio-b39e1df06edb0dd5e1faf225921ac555de05818a1d050fba677439f8d7726d60 WatchSource:0}: Error finding container b39e1df06edb0dd5e1faf225921ac555de05818a1d050fba677439f8d7726d60: Status 404 returned error can't find the container with id b39e1df06edb0dd5e1faf225921ac555de05818a1d050fba677439f8d7726d60 Oct 30 20:03:50.559960 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.559929 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j68lw/must-gather-4dlfv"] Oct 30 20:03:50.561092 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:50.561074 4789 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:03:51.013595 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:51.013557 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j68lw/must-gather-4dlfv" event={"ID":"f638db36-367a-4e15-b6bf-330132786526","Type":"ContainerStarted","Data":"b39e1df06edb0dd5e1faf225921ac555de05818a1d050fba677439f8d7726d60"} Oct 30 20:03:52.024774 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:52.024733 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j68lw/must-gather-4dlfv" event={"ID":"f638db36-367a-4e15-b6bf-330132786526","Type":"ContainerStarted","Data":"fced0a7fa3971edb7750d482b4686ad66779bee68219783a67647577f3891a75"} Oct 30 20:03:52.087813 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:52.087779 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z"] Oct 30 20:03:52.120038 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:52.120014 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2"] Oct 30 20:03:52.134495 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:52.134445 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed74471-qvs6z"] Oct 30 20:03:52.136226 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:52.136191 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5cabe1d-mhjv2"] Oct 30 20:03:53.032921 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:53.032886 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j68lw/must-gather-4dlfv" event={"ID":"f638db36-367a-4e15-b6bf-330132786526","Type":"ContainerStarted","Data":"b58490a9fb2f2a8d74305a6c67ced43f869ddeb08b6d3d7af333d8eab22a2355"} Oct 30 20:03:53.155222 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:53.155155 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j68lw/must-gather-4dlfv" podStartSLOduration=2.843898615 podStartE2EDuration="4.15513769s" totalImagesPullingTime="1.311239075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:53.154362094 +0000 UTC m=+4669.656272580" watchObservedRunningTime="2026-10-30 20:03:53.15513769 +0000 UTC m=+4669.657048176" Oct 30 20:03:53.667419 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:53.667384 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6bb431c-f448-49a9-9557-0a98da55fbb9" path="/var/lib/kubelet/pods/f6bb431c-f448-49a9-9557-0a98da55fbb9/volumes" Oct 30 20:03:53.667814 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:53.667801 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d256cb-f374-476d-808a-8d2bb9c8b794" path="/var/lib/kubelet/pods/f6d256cb-f374-476d-808a-8d2bb9c8b794/volumes" Oct 30 20:03:59.132228 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:03:59.116714 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vnc44_9cdd95a5-32ae-4c7a-a067-bc40ad4bc826/global-pull-secret-syncer/0.log" Oct 30 20:04:00.295229 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:00.294706 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-54zh4_497867c2-a03d-4717-b845-1d0ef71495d5/connection-checker/0.log" Oct 30 20:04:00.724228 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:00.723479 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bzdtx_41bb2e81-6a7b-434e-a6c4-184beef52d40/connection-checker/0.log" Oct 30 20:04:01.155906 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:01.155810 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fj8mt_8c417afb-ca2e-4706-8004-f6651dd11f16/connection-checker/0.log" Oct 30 20:04:02.055931 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:02.055891 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q"] Oct 30 20:04:02.086157 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:02.086122 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8"] Oct 30 20:04:02.102315 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:02.102279 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2ab9f3a-rrp7q"] Oct 30 20:04:02.105271 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:02.105246 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbaa2a2-jjrz8"] Oct 30 20:04:02.270311 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:02.270278 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rs2ft_150b998b-9608-4846-ac54-8a8ec8a7c620/konnectivity-agent/0.log" Oct 30 20:04:03.104566 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:03.104536 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-46.us-west-2.compute.internal_29dce357bba57b17e7ff4a65afbed63d/haproxy/0.log" Oct 30 20:04:03.668763 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:03.668728 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0" path="/var/lib/kubelet/pods/26b4b59f-8ecb-40c3-b4d7-4dcfd13a55f0/volumes" Oct 30 20:04:03.669255 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:03.669226 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc" path="/var/lib/kubelet/pods/bb8c9f97-03a7-4f54-a2f4-5e009d16a5fc/volumes" Oct 30 20:04:05.546343 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:05.546299 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:05.990203 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:05.990160 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:07.351383 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:07.351338 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:07.774170 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:07.774075 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:07.918187 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:07.918131 4789 scope.go:122] "RemoveContainer" containerID="51d37c786d4e453f8c5802b7ec6bbdfd268046240a33cdb932c4b5c2fc6243f0" Oct 30 20:04:07.927811 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:07.927790 4789 scope.go:122] "RemoveContainer" containerID="b281eba6ce91dbe4e5d29900dddd7223852b29a3168166973ad24f246ebdb5e2" Oct 30 20:04:07.936468 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:07.936443 4789 scope.go:122] "RemoveContainer" containerID="b191ff4c5a88963d25307be39b4077bd92caca13ae61fede2ced07ddd7333cc7" Oct 30 20:04:08.088271 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:08.087673 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j"] Oct 30 20:04:08.133867 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:08.133807 4789 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw"] Oct 30 20:04:08.136567 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:08.136541 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd5c480a-whv9j"] Oct 30 20:04:08.138910 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:08.138884 4789 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6a7660-nfhrw"] Oct 30 20:04:09.107074 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.107044 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6vfk4_36e0b317-a46f-48dc-9c68-501c0d36bef9/manager/0.log" Oct 30 20:04:09.266128 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.266094 4789 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l"] Oct 30 20:04:09.272956 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.272931 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.326036 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.325995 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l"] Oct 30 20:04:09.340296 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.340269 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthpl\" (UniqueName: \"kubernetes.io/projected/20c25f29-4864-46f4-9af1-7a43eec3a761-kube-api-access-gthpl\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.340436 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.340313 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-podres\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.340436 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.340428 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-lib-modules\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.340524 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.340447 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-sys\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.340524 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.340474 4789 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-proc\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441086 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441053 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-lib-modules\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441088 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-sys\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441127 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-proc\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441169 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthpl\" (UniqueName: \"kubernetes.io/projected/20c25f29-4864-46f4-9af1-7a43eec3a761-kube-api-access-gthpl\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441197 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-sys\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441233 4789 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-podres\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441243 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-lib-modules\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441291 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441270 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-proc\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.441677 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.441346 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20c25f29-4864-46f4-9af1-7a43eec3a761-podres\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.500688 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.500657 4789 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthpl\" (UniqueName: \"kubernetes.io/projected/20c25f29-4864-46f4-9af1-7a43eec3a761-kube-api-access-gthpl\") pod \"perf-node-gather-daemonset-qkw8l\" (UID: \"20c25f29-4864-46f4-9af1-7a43eec3a761\") " pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.586375 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.586345 4789 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:09.671171 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.671124 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d6be7de-7a61-4f00-97ca-e157806ccaef" path="/var/lib/kubelet/pods/8d6be7de-7a61-4f00-97ca-e157806ccaef/volumes" Oct 30 20:04:09.671625 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.671608 4789 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bf77b8-a6c1-44aa-9003-2212327d0a04" path="/var/lib/kubelet/pods/c0bf77b8-a6c1-44aa-9003-2212327d0a04/volumes" Oct 30 20:04:09.974192 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:09.974153 4789 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l"] Oct 30 20:04:09.980805 ip-10-0-136-46 kubenswrapper[4789]: W1030 20:04:09.980779 4789 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20c25f29_4864_46f4_9af1_7a43eec3a761.slice/crio-0c8f78c39e3b97820c9750386e3659039b7d68976f9867b91bb9fe99defd4779 WatchSource:0}: Error finding container 0c8f78c39e3b97820c9750386e3659039b7d68976f9867b91bb9fe99defd4779: Status 404 returned error can't find the container with id 0c8f78c39e3b97820c9750386e3659039b7d68976f9867b91bb9fe99defd4779 Oct 30 20:04:10.226889 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:10.226795 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" event={"ID":"20c25f29-4864-46f4-9af1-7a43eec3a761","Type":"ContainerStarted","Data":"f08c749174adffd72fb33e5ec49495e83807d47cb3da386c08ffe2eac07d67f9"} Oct 30 20:04:10.227398 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:10.226902 4789 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" event={"ID":"20c25f29-4864-46f4-9af1-7a43eec3a761","Type":"ContainerStarted","Data":"0c8f78c39e3b97820c9750386e3659039b7d68976f9867b91bb9fe99defd4779"} Oct 30 20:04:10.227398 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:10.226941 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:10.352223 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:10.352159 4789 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" podStartSLOduration=1.352142797 podStartE2EDuration="1.352142797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:10.350452501 +0000 UTC m=+4686.852362987" watchObservedRunningTime="2026-10-30 20:04:10.352142797 +0000 UTC m=+4686.854053283" Oct 30 20:04:10.731856 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:10.731824 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kkqxp_05644db3-a8f4-4fad-8b6a-0713ac0c8f01/manager/0.log" Oct 30 20:04:11.530931 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:11.530895 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:11.612257 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:11.612221 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rxf5s_b4d866b2-7844-4f4a-ba79-31d75d4d8e76/manager/0.log" Oct 30 20:04:12.096616 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:12.096582 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:13.304936 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:13.304902 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:13.797785 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:13.797745 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:16.244831 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:16.244796 4789 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j68lw/perf-node-gather-daemonset-qkw8l" Oct 30 20:04:17.237664 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:17.237633 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:17.548683 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:17.548604 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-lmhht_f7666d1a-d160-4cc8-a277-208408706b6d/cluster-monitoring-operator/0.log" Oct 30 20:04:17.869891 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:17.869808 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-txf57_0f4b3eb6-8ea7-405a-8706-b35aaa6ee132/manager/0.log" Oct 30 20:04:18.199759 ip-10-0-136-46 kubenswrapper[4789]: E1030 20:04:18.199702 4789 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2ckq9" containerName="extractor" Oct 30 20:04:18.676552 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:18.676437 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:19.347913 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:19.347876 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-c57ns_8a9da57a-6499-4c34-832d-a0cf66b01a3d/manager/0.log" Oct 30 20:04:24.406335 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:24.406306 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t57bd_77e7d96a-fa2f-44c3-9d99-81e562a2292e/node-exporter/0.log" Oct 30 20:04:24.815381 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:24.815346 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t57bd_77e7d96a-fa2f-44c3-9d99-81e562a2292e/kube-rbac-proxy/0.log" Oct 30 20:04:24.955843 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:24.955812 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df7c6869c-nrlk4_4cae15c3-20da-4a51-8b5c-3386568b4d72/kube-auth-proxy/0.log" Oct 30 20:04:25.125754 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:25.125677 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df7c6869c-nrlk4_4cae15c3-20da-4a51-8b5c-3386568b4d72/kube-auth-proxy/0.log" Oct 30 20:04:25.224821 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:25.224795 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t57bd_77e7d96a-fa2f-44c3-9d99-81e562a2292e/init-textfile/0.log" Oct 30 20:04:25.281690 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:25.281661 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp_45bae6ce-241c-4f8d-b724-6ea46abc05e6/istio-proxy/0.log" Oct 30 20:04:25.481060 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:25.481032 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp_45bae6ce-241c-4f8d-b724-6ea46abc05e6/istio-proxy/0.log" Oct 30 20:04:25.931159 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:25.931080 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5978b965c7-kbthb_0605710c-1a4a-47f2-a7bc-4079a97e9db4/router/0.log" Oct 30 20:04:26.160070 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:26.160037 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5978b965c7-kbthb_0605710c-1a4a-47f2-a7bc-4079a97e9db4/router/0.log" Oct 30 20:04:30.594919 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:30.594891 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-298qb_f628c55a-1a6a-475e-b328-2ae83339e484/prometheus-operator-admission-webhook/0.log" Oct 30 20:04:32.400393 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:32.400363 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6vfk4_36e0b317-a46f-48dc-9c68-501c0d36bef9/manager/0.log" Oct 30 20:04:32.797786 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:32.797742 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-6vfk4_36e0b317-a46f-48dc-9c68-501c0d36bef9/manager/0.log" Oct 30 20:04:33.714962 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:33.714928 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kkqxp_05644db3-a8f4-4fad-8b6a-0713ac0c8f01/manager/0.log" Oct 30 20:04:33.778513 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:33.778484 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6df7c6869c-nrlk4_4cae15c3-20da-4a51-8b5c-3386568b4d72/kube-auth-proxy/0.log" Oct 30 20:04:34.105564 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:34.105471 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kkqxp_05644db3-a8f4-4fad-8b6a-0713ac0c8f01/manager/0.log" Oct 30 20:04:34.151932 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:34.151896 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rxf5s_b4d866b2-7844-4f4a-ba79-31d75d4d8e76/manager/0.log" Oct 30 20:04:34.192589 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:34.192560 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-77b95bc695-mmwdp_45bae6ce-241c-4f8d-b724-6ea46abc05e6/istio-proxy/0.log" Oct 30 20:04:34.496698 ip-10-0-136-46 kubenswrapper[4789]: I1030 20:04:34.496652 4789 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rxf5s_b4d866b2-7844-4f4a-ba79-31d75d4d8e76/manager/0.log"